Jul 07 21:59:02.386548 ip-10-0-130-137 systemd[1]: Starting Kubernetes Kubelet... Jul 07 21:59:02.792959 ip-10-0-130-137 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 21:59:02.792959 ip-10-0-130-137 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 21:59:02.792959 ip-10-0-130-137 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 21:59:02.792959 ip-10-0-130-137 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 21:59:02.792959 ip-10-0-130-137 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 21:59:02.793657 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.792974 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 21:59:02.799526 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799505 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 21:59:02.799526 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799525 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 21:59:02.799526 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799529 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799533 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799536 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799540 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799543 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799547 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799552 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799555 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799559 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799561 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799564 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799568 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799571 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799574 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799576 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799579 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799581 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799584 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799586 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799589 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 21:59:02.799630 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799591 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799594 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799596 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799599 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799601 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799604 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799606 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799609 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799611 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799614 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799616 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799619 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799621 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799624 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799627 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799630 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799634 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799636 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799639 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 21:59:02.800147 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799642 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799644 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799647 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799651 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799653 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799656 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799658 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799660 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799663 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799665 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799668 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799670 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799674 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799678 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799682 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799685 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799688 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799691 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799693 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 21:59:02.800639 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799696 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799699 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799702 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799704 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799707 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799710 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799712 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799715 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799719 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799722 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799724 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799730 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799732 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799735 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799737 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799740 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799742 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799745 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799747 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799750 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 21:59:02.801133 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799753 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799755 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799757 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799767 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799770 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799773 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799775 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799778 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799780 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799783 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799785 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799788 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799790 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799793 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799795 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799798 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799800 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799803 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 21:59:02.801633 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.799805 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 21:59:02.802093 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802078 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 21:59:02.802093 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802092 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802096 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802099 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802102 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802105 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802109 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802112 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802114 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802117 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802119 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802122 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802124 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802126 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802129 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802131 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802134 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802136 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802138 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802141 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802144 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 21:59:02.802150 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802146 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802148 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802151 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802154 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802157 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802159 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802162 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802165 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802168 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802170 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802189 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802192 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802194 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802197 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802199 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802202 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802205 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802208 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802211 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802213 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 21:59:02.802655 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802216 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802221 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802225 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802227 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802230 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802232 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802235 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802237 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802240 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802242 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802245 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802247 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802250 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802252 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802256 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802258 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802261 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802264 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802266 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802269 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 21:59:02.803135 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802271 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802275 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802278 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802281 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802283 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802285 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802289 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802292 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802295 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802298 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802301 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802304 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802306 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802309 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802311 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802314 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802316 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802318 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802321 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 21:59:02.803776 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802323 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802326 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802328 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802330 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802333 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802335 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802338 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802343 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802345 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802348 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802352 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802355 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802358 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802361 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802364 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802366 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802369 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802372 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802375 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802805 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 21:59:02.804271 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802810 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802813 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802816 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802820 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802823 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802825 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802828 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802830 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802833 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802835 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802838 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802841 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802843 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802848 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802851 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802854 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802857 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802860 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802862 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 21:59:02.804754 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802866 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802869 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802872 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802875 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802878 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802881 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802884 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802886 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802889 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802892 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802894 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802897 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802900 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802902 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802905 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802907 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802910 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802913 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802915 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802918 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 21:59:02.805222 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802920 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802923 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802926 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802928 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802931 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802933 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802936 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802938 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802941 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802945 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802948 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802951 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802954 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802957 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802960 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802962 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802965 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802968 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802970 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802972 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 21:59:02.805702 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802975 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802979 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802982 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802985 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802987 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802990 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802992 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802995 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.802997 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803000 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803009 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803012 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803014 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803017 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803020 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803024 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803026 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803030 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803032 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803035 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 21:59:02.806196 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803038 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803040 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803043 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803046 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803048 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803051 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803053 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803056 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803059 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803061 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803064 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803066 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803069 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803074 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803076 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803078 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803081 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803084 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803086 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803184 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803193 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 21:59:02.806668 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803199 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803204 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803216 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803219 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803223 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803227 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803230 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803234 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803237 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803240 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803243 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803246 2565 flags.go:64] FLAG: --cgroup-root="" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803249 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803252 2565 flags.go:64] FLAG: --client-ca-file="" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803255 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803260 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803264 2565 flags.go:64] FLAG: --cluster-domain="" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803267 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803271 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803274 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803277 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803281 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803284 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 21:59:02.807184 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803287 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803291 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803294 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803296 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803299 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803302 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803305 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803309 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803312 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803315 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803318 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803321 2565 flags.go:64] FLAG: --enable-server="true" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803328 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803332 2565 flags.go:64] FLAG: --event-burst="100" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803335 2565 flags.go:64] FLAG: --event-qps="50" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803338 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803341 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803344 2565 flags.go:64] FLAG: --eviction-hard="" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803348 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803351 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803354 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803357 2565 flags.go:64] FLAG: --eviction-soft="" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803359 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803363 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803367 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 21:59:02.807740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803370 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803373 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803376 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803379 2565 flags.go:64] FLAG: --feature-gates="" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803383 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803386 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803389 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803392 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803396 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803399 2565 flags.go:64] FLAG: --help="false" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803402 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-130-137.ec2.internal" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803405 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803408 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803410 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803414 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803417 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803420 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803423 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803426 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803429 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803432 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803435 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803438 2565 flags.go:64] FLAG: --kube-reserved="" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803441 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 21:59:02.808433 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803443 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803446 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803449 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803452 2565 flags.go:64] FLAG: --lock-file="" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803454 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803457 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803460 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803467 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803470 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803473 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803476 2565 flags.go:64] FLAG: --logging-format="text" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803478 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803482 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803484 2565 flags.go:64] FLAG: --manifest-url="" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803487 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803492 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803495 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803499 2565 flags.go:64] FLAG: --max-pods="110" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803502 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803505 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803507 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803510 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803513 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803516 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803520 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 21:59:02.809018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803528 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803531 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803534 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803537 2565 flags.go:64] FLAG: --pod-cidr="" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803540 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803547 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803550 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803553 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803556 2565 flags.go:64] FLAG: --port="10250" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803559 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803562 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09ed5254d72292f58" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803565 2565 flags.go:64] FLAG: --qos-reserved="" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803568 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803571 2565 flags.go:64] FLAG: --register-node="true" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803574 2565 flags.go:64] FLAG: --register-with-taints="" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803578 2565 flags.go:64] FLAG: --registry-burst="10" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803581 2565 flags.go:64] FLAG: --registry-qps="5" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803584 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803587 2565 flags.go:64] FLAG: --reserved-memory="" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803590 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803593 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803596 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803599 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803602 2565 flags.go:64] FLAG: --runonce="false" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803604 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 21:59:02.809664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803608 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803610 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803613 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803616 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803619 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803622 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803625 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803628 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803630 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803634 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803637 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803640 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803643 2565 flags.go:64] FLAG: --system-cgroups="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803646 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803651 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803654 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803657 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803661 2565 flags.go:64] FLAG: --tls-min-version="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803664 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803666 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803669 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803672 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803677 2565 flags.go:64] FLAG: --v="2" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803681 2565 flags.go:64] FLAG: --version="false" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803685 2565 flags.go:64] FLAG: --vmodule="" Jul 07 21:59:02.810283 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803689 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.803692 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803817 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803822 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803825 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803829 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803833 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803836 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803839 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803842 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803846 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803849 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803851 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803854 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803856 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803859 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803861 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803864 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803867 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803870 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 21:59:02.810895 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803873 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803875 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803878 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803880 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803883 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803885 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803888 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803891 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803893 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803897 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803900 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803902 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803904 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803907 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803909 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803912 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803915 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803919 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803923 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 21:59:02.811402 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803926 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803929 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803932 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803935 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803938 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803941 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803943 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803946 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803948 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803951 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803953 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803956 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803959 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803961 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803970 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803973 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803975 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803977 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803980 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803982 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 21:59:02.811880 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803985 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803987 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803991 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803993 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803996 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.803999 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804001 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804004 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804006 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804008 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804013 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804017 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804020 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804022 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804025 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804027 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804030 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804033 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804035 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 21:59:02.812607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804037 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804040 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804042 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804045 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804047 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804050 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804052 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804055 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804062 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804065 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804068 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804070 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804073 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804075 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804077 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804081 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804084 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804086 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804089 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804091 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 21:59:02.813499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804094 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 21:59:02.814045 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804096 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 21:59:02.814045 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.804099 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 21:59:02.814045 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.805068 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 21:59:02.814479 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.814454 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 21:59:02.814522 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.814480 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 21:59:02.814522 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.814496 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 21:59:02.814522 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.814502 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814566 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814572 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814575 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814579 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814581 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814584 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814587 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814589 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814592 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814594 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814597 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814600 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814602 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814605 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814607 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814611 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 21:59:02.814607 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814614 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814617 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814620 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814622 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814627 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814633 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814637 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814639 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814642 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814645 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814647 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814650 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814653 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814655 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814658 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814660 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814663 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814666 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814669 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 21:59:02.815011 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814671 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814674 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814676 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814678 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814681 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814683 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814686 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814689 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814692 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814694 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814697 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814700 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814703 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814705 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814708 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814710 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814713 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814716 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814718 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814721 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 21:59:02.815499 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814723 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814726 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814728 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814730 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814733 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814736 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814738 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814741 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814745 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814748 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814751 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814754 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814756 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814759 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814762 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814764 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814767 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814769 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814772 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 21:59:02.815991 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814775 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814777 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814779 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814782 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814784 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814788 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814790 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814793 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814796 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814798 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814801 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814803 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814807 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814811 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814814 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814817 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814819 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814822 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814825 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814828 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 21:59:02.816474 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814830 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814833 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814835 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814837 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814840 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.814845 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814974 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814979 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814982 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814985 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814988 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814991 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814994 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.814997 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815000 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 21:59:02.816963 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815002 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815005 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815007 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815011 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815014 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815017 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815020 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815023 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815026 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815029 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815031 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815034 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815036 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815039 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815041 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815044 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815046 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815049 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815051 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815054 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 21:59:02.817399 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815056 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815060 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815062 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815065 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815067 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815070 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815072 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815075 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815077 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815080 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815082 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815085 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815087 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815090 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815093 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815095 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815099 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815101 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815105 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815107 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 21:59:02.817889 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815110 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815113 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815115 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815118 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815122 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815125 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815128 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815131 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815134 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815136 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815139 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815141 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815143 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815146 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815149 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815152 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815154 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815156 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815159 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815161 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 21:59:02.818394 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815164 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815166 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815169 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815187 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815190 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815194 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815197 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815200 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815203 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815206 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815209 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815213 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815215 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815219 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815223 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815226 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815228 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815231 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815234 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 21:59:02.818870 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815236 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815239 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815241 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815244 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815247 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815249 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815252 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815254 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815257 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815259 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:02.815262 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.815267 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.815410 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 21:59:02.819348 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.818843 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 21:59:02.820518 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.820495 2565 server.go:1019] "Starting client certificate rotation" Jul 07 21:59:02.820633 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.820612 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 21:59:02.820663 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.820657 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 21:59:02.823329 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.823312 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 21:59:02.826587 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.826563 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 21:59:02.836447 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.836415 2565 log.go:25] "Validated CRI v1 runtime API" Jul 07 21:59:02.840772 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.840749 2565 log.go:25] "Validated CRI v1 image API" Jul 07 21:59:02.841710 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.841691 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 21:59:02.844159 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.844136 2565 fs.go:135] Filesystem UUIDs: map[4a712b4f-06d0-43e4-9809-7ab6169cbc00:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 93edcaab-cf05-40a5-a2f4-4c1b38008ffd:/dev/nvme0n1p3] Jul 07 21:59:02.844271 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.844157 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 21:59:02.850293 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.850162 2565 manager.go:216] Machine: {Timestamp:2026-07-07 21:59:02.848759398 +0000 UTC m=+0.343180797 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3004475 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21699cd13d6b451747d3ac0e0477b2 SystemUUID:ec21699c-d13d-6b45-1747-d3ac0e0477b2 BootID:1d3e47af-6128-4775-9b9a-76418f469849 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9d:bd:22:58:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9d:bd:22:58:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:d7:23:f6:29:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 21:59:02.850293 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.850286 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 21:59:02.850406 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.850376 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 21:59:02.853895 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.853859 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 21:59:02.854039 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.853897 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 21:59:02.855257 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.855247 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 21:59:02.855304 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.855260 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 21:59:02.855304 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.855267 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 21:59:02.856037 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.856027 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 21:59:02.856739 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.856723 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 21:59:02.858025 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.858013 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 21:59:02.861863 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.861850 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 07 21:59:02.862194 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.862168 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 21:59:02.862233 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.862219 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 07 21:59:02.862233 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.862230 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 21:59:02.862311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.862244 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 21:59:02.862311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.862253 2565 kubelet.go:398] "Adding apiserver pod source" Jul 07 21:59:02.862311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.862263 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 21:59:02.863556 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.863524 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 21:59:02.863661 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.863568 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 21:59:02.866719 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.866698 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 21:59:02.869132 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.869074 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 21:59:02.869132 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.869106 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 21:59:02.871086 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871073 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871091 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871097 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871104 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871113 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871118 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871127 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 21:59:02.871131 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871132 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 21:59:02.871345 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871140 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 21:59:02.871345 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871146 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 21:59:02.871345 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871155 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 21:59:02.871345 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.871160 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 21:59:02.872056 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.872044 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 21:59:02.872056 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.872055 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 21:59:02.874713 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.874650 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 21:59:02.874713 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.874675 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 21:59:02.875490 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.875469 2565 server.go:1268] "Started kubelet" Jul 07 21:59:02.875605 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.875552 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 21:59:02.875605 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.875546 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 21:59:02.875685 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.875608 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 21:59:02.876380 ip-10-0-130-137 systemd[1]: Started Kubernetes Kubelet. Jul 07 21:59:02.878018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.877999 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 21:59:02.878209 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.878193 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 07 21:59:02.881284 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.881265 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 21:59:02.886252 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.886236 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 21:59:02.886330 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.886251 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 21:59:02.886993 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.886422 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:02.886993 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.886807 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 21:59:02.887158 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887019 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 21:59:02.887158 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887051 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 21:59:02.887158 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887063 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 21:59:02.887325 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887232 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 07 21:59:02.887325 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887240 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 07 21:59:02.887476 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887453 2565 factory.go:55] Registering systemd factory Jul 07 21:59:02.887476 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887473 2565 factory.go:223] Registration of the systemd container factory successfully Jul 07 21:59:02.887646 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.886649 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-137.ec2.internal.18c020d2f5a9129f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-137.ec2.internal,UID:ip-10-0-130-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-137.ec2.internal,},FirstTimestamp:2026-07-07 21:59:02.875435679 +0000 UTC m=+0.369857126,LastTimestamp:2026-07-07 21:59:02.875435679 +0000 UTC m=+0.369857126,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-137.ec2.internal,}" Jul 07 21:59:02.887797 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887781 2565 factory.go:153] Registering CRI-O factory Jul 07 21:59:02.887859 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887801 2565 factory.go:223] Registration of the crio container factory successfully Jul 07 21:59:02.887859 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887850 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 21:59:02.887931 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887873 2565 factory.go:103] Registering Raw factory Jul 07 21:59:02.887931 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.887887 2565 manager.go:1196] Started watching for new ooms in manager Jul 07 21:59:02.888378 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.888360 2565 manager.go:349] Starting recovery of all containers Jul 07 21:59:02.898966 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.898931 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 21:59:02.901601 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.901580 2565 manager.go:354] Recovery completed Jul 07 21:59:02.904036 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.904013 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t7wmq" Jul 07 21:59:02.905807 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.905793 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:02.907598 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.907575 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:02.907671 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.907609 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:02.907671 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.907619 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:02.908235 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.908217 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 21:59:02.908235 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.908231 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 21:59:02.908370 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.908239 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:02.908370 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.908249 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 07 21:59:02.909005 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.908985 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:02.909088 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.909018 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:02.909088 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.909031 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:02.909505 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.909493 2565 policy_none.go:49] "None policy: Start" Jul 07 21:59:02.909547 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.909510 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 21:59:02.909547 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.909520 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 21:59:02.910521 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.910511 2565 policy_none.go:47] "Start" Jul 07 21:59:02.911082 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.911007 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-137.ec2.internal.18c020d2f793c8d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-137.ec2.internal,UID:ip-10-0-130-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-137.ec2.internal,},FirstTimestamp:2026-07-07 21:59:02.907594963 +0000 UTC m=+0.402016365,LastTimestamp:2026-07-07 21:59:02.907594963 +0000 UTC m=+0.402016365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-137.ec2.internal,}" Jul 07 21:59:02.911534 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.911517 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t7wmq" Jul 07 21:59:02.916546 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.916454 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-137.ec2.internal.18c020d2f7941114 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-137.ec2.internal,UID:ip-10-0-130-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-137.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-137.ec2.internal,},FirstTimestamp:2026-07-07 21:59:02.90761346 +0000 UTC m=+0.402034858,LastTimestamp:2026-07-07 21:59:02.90761346 +0000 UTC m=+0.402034858,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-137.ec2.internal,}" Jul 07 21:59:02.916663 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.916579 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 21:59:02.950635 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.950619 2565 manager.go:337] "Starting Device Plugin manager" Jul 07 21:59:02.950728 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.950656 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 21:59:02.950728 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.950666 2565 server.go:87] "Starting device plugin registration server" Jul 07 21:59:02.950960 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.950947 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 21:59:02.950992 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.950964 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 21:59:02.951055 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.951036 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 21:59:02.951129 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.951119 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 21:59:02.951129 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:02.951129 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 21:59:02.951807 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.951784 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 21:59:02.951910 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:02.951844 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.051730 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.051645 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:03.052928 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.052890 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:03.052928 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.052924 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:03.053035 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.052938 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:03.053035 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.052965 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.059161 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.059139 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.059286 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.059170 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-137.ec2.internal\": node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.065456 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.065419 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 21:59:03.066853 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.066839 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 21:59:03.066949 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.066872 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 21:59:03.066949 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.066893 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 21:59:03.066949 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.066926 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 21:59:03.069444 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.069422 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 21:59:03.084359 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.084323 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.167768 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.167729 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal"] Jul 07 21:59:03.167843 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.167813 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:03.169612 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.169591 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:03.169708 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.169625 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:03.169708 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.169635 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:03.170883 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.170869 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:03.170990 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.170976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.171027 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171005 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:03.171702 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171674 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:03.171832 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171708 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:03.171832 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171716 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:03.171832 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171723 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:03.171832 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171742 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:03.171832 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.171755 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:03.172879 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.172865 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.172951 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.172890 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 21:59:03.174192 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.174163 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientMemory" Jul 07 21:59:03.174288 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.174261 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 21:59:03.174288 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.174278 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeHasSufficientPID" Jul 07 21:59:03.185135 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.185107 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.189517 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.189492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/223e82db7a0ce00bd93fd50f651c4d28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal\" (UID: \"223e82db7a0ce00bd93fd50f651c4d28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.189616 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.189533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/223e82db7a0ce00bd93fd50f651c4d28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal\" (UID: \"223e82db7a0ce00bd93fd50f651c4d28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.189616 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.189562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5456099913828cc9faa58b08483592f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-137.ec2.internal\" (UID: \"b5456099913828cc9faa58b08483592f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.194901 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.194881 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-137.ec2.internal\" not found" node="ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.199800 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.199777 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-137.ec2.internal\" not found" node="ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.285512 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.285471 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.289850 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.289830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/223e82db7a0ce00bd93fd50f651c4d28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal\" (UID: \"223e82db7a0ce00bd93fd50f651c4d28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.289935 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.289859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/223e82db7a0ce00bd93fd50f651c4d28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal\" (UID: \"223e82db7a0ce00bd93fd50f651c4d28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.289935 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.289875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5456099913828cc9faa58b08483592f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-137.ec2.internal\" (UID: \"b5456099913828cc9faa58b08483592f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.289935 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.289923 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5456099913828cc9faa58b08483592f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-137.ec2.internal\" (UID: \"b5456099913828cc9faa58b08483592f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.290059 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.289940 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/223e82db7a0ce00bd93fd50f651c4d28-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal\" (UID: \"223e82db7a0ce00bd93fd50f651c4d28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.290059 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.289970 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/223e82db7a0ce00bd93fd50f651c4d28-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal\" (UID: \"223e82db7a0ce00bd93fd50f651c4d28\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.385655 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.385553 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.486101 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.486071 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:03.498506 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.498476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.503958 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:03.503936 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" Jul 07 21:59:03.586290 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:03.586245 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:05.189609 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:05.189573 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:05.290038 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:05.289987 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-137.ec2.internal\" not found" Jul 07 21:59:05.322374 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.322336 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 21:59:05.322986 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.322518 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 21:59:05.369189 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.369157 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 21:59:05.390174 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.390136 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 21:59:05.390174 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.390163 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" Jul 07 21:59:05.400745 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.400671 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 21:59:05.401113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.401090 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 21:59:05.402752 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.402734 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" Jul 07 21:59:05.411892 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.411870 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 21:59:05.418997 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.418964 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 21:54:04 +0000 UTC" deadline="2028-03-30 15:38:21.783064584 +0000 UTC" Jul 07 21:59:05.418997 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.418995 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15161h39m16.364072883s" Jul 07 21:59:05.441581 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.441550 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9wblb" Jul 07 21:59:05.450964 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.450932 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9wblb" Jul 07 21:59:05.514524 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:05.514483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5456099913828cc9faa58b08483592f.slice/crio-4ac4a707db36fa5cdb661741017bdf03ece879fad980f749aca60c7e831fe6ae WatchSource:0}: Error finding container 4ac4a707db36fa5cdb661741017bdf03ece879fad980f749aca60c7e831fe6ae: Status 404 returned error can't find the container with id 4ac4a707db36fa5cdb661741017bdf03ece879fad980f749aca60c7e831fe6ae Jul 07 21:59:05.515156 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:05.515134 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223e82db7a0ce00bd93fd50f651c4d28.slice/crio-e78e1b07aa17a49d8669a70879026d4316ec0b9e6d91d33af2e4dd5900ed4c1d WatchSource:0}: Error finding container e78e1b07aa17a49d8669a70879026d4316ec0b9e6d91d33af2e4dd5900ed4c1d: Status 404 returned error can't find the container with id e78e1b07aa17a49d8669a70879026d4316ec0b9e6d91d33af2e4dd5900ed4c1d Jul 07 21:59:05.520939 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.520923 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 21:59:05.573399 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.573345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" event={"ID":"223e82db7a0ce00bd93fd50f651c4d28","Type":"ContainerStarted","Data":"e78e1b07aa17a49d8669a70879026d4316ec0b9e6d91d33af2e4dd5900ed4c1d"} Jul 07 21:59:05.574378 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.574351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" event={"ID":"b5456099913828cc9faa58b08483592f","Type":"ContainerStarted","Data":"4ac4a707db36fa5cdb661741017bdf03ece879fad980f749aca60c7e831fe6ae"} Jul 07 21:59:05.948079 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.948050 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 21:59:05.956379 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:05.956353 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 21:59:06.270996 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.270905 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 21:59:06.366818 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.366785 2565 apiserver.go:52] "Watching apiserver" Jul 07 21:59:06.372794 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.372761 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 21:59:06.373382 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.373361 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cz4t2","kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9","openshift-cluster-node-tuning-operator/tuned-pkwtz","openshift-dns/node-resolver-7k7p6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal","openshift-multus/multus-additional-cni-plugins-jnfnp","openshift-multus/multus-w8gkx","kube-system/konnectivity-agent-h8n52","openshift-image-registry/node-ca-8m9l6","openshift-multus/network-metrics-daemon-4zzjj","openshift-network-diagnostics/network-check-target-qf59h","openshift-network-operator/iptables-alerter-pgzth","openshift-ovn-kubernetes/ovnkube-node-fm898"] Jul 07 21:59:06.375318 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.374868 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.376143 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.376123 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.378192 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.378167 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.378319 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.378289 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.378540 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.378522 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 21:59:06.379002 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.378987 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.379295 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.379277 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lqmfj\"" Jul 07 21:59:06.379552 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.379537 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.380122 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.380104 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 21:59:06.380238 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.380128 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.380342 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.380321 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w2rqs\"" Jul 07 21:59:06.380503 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.380480 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 21:59:06.380736 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.380714 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.380799 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.380789 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 21:59:06.381343 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.381075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-78kmp\"" Jul 07 21:59:06.381343 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.381108 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.381343 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.381320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.382632 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.382598 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.382945 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.382927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.384470 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.384448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.385673 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.385654 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.385802 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.385771 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:06.386483 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.386463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pwzmm\"" Jul 07 21:59:06.386663 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.386640 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rvnd6\"" Jul 07 21:59:06.387047 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.386855 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 21:59:06.387047 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.386871 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.387183 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387132 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.387183 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387136 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.387646 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387322 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.387646 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ntss6\"" Jul 07 21:59:06.387646 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387518 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.387646 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387583 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 21:59:06.387868 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387657 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 21:59:06.387868 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387752 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 21:59:06.387868 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387847 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 21:59:06.388002 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387889 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8zrx9\"" Jul 07 21:59:06.388002 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.387997 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.388191 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.388048 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 21:59:06.388388 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.388228 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 21:59:06.388388 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.388296 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.389565 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.389200 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.391621 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.391601 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 21:59:06.391721 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.391603 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 21:59:06.391721 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.391609 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 21:59:06.391944 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.391924 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ff58k\"" Jul 07 21:59:06.392045 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.391989 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bs8lz\"" Jul 07 21:59:06.392355 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.392339 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 21:59:06.397093 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.397069 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:06.397216 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.397162 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:06.398479 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.398456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:06.398586 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.398534 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:06.400383 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-kubelet\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.400507 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbw7b\" (UniqueName: \"kubernetes.io/projected/c0b7e5bc-a55f-4475-a9ce-a3f386594731-kube-api-access-dbw7b\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.400507 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-sys-fs\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.400507 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88226ee8-c447-4230-a118-614eb688c5f9-ovn-node-metrics-cert\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.400507 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/402eead4-ad2d-4876-9e68-54be18236c77-cni-binary-copy\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.400806 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-modprobe-d\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.400806 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysconfig\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.400806 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-lib-modules\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.400806 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:06.400806 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-k8s-cni-cncf-io\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-cni-multus\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k594k\" (UniqueName: \"kubernetes.io/projected/402eead4-ad2d-4876-9e68-54be18236c77-kube-api-access-k594k\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphv9\" (UniqueName: \"kubernetes.io/projected/f747861a-b056-485e-93f3-0d4491053813-kube-api-access-cphv9\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzv9\" (UniqueName: \"kubernetes.io/projected/48345d08-5995-4c23-a63c-c55b32e8d7cd-kube-api-access-rrzv9\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-systemd-units\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.400983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/402eead4-ad2d-4876-9e68-54be18236c77-multus-daemon-config\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cnibin\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401047 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-slash\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-socket-dir-parent\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-kubernetes\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.401110 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-host\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-device-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401164 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-run-netns\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-var-lib-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-node-log\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-multus-certs\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-os-release\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysctl-conf\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-registration-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401543 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-ovnkube-script-lib\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401571 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.401614 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-hostroot\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-etc-kubernetes\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-run\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401668 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f747861a-b056-485e-93f3-0d4491053813-etc-tuned\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a41a985-5a2d-425b-8723-2fffc40a1754-agent-certs\") pod \"konnectivity-agent-h8n52\" (UID: \"8a41a985-5a2d-425b-8723-2fffc40a1754\") " pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ede2f80-924e-4425-92f0-6bc4ac661c93-hosts-file\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-log-socket\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-cni-bin\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmf2\" (UniqueName: \"kubernetes.io/projected/88226ee8-c447-4230-a118-614eb688c5f9-kube-api-access-ftmf2\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9b71ef0-3555-4870-9392-4955495398c0-dbus\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-systemd\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-cni-netd\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a41a985-5a2d-425b-8723-2fffc40a1754-konnectivity-ca\") pod \"konnectivity-agent-h8n52\" (UID: \"8a41a985-5a2d-425b-8723-2fffc40a1754\") " pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.402311 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nspm\" (UniqueName: \"kubernetes.io/projected/17749b82-0e66-4e1d-a410-a6f414f06ff1-kube-api-access-4nspm\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.401989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-sys\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48345d08-5995-4c23-a63c-c55b32e8d7cd-serviceca\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac401dbf-9a67-4224-884c-6a81c2e37a8e-iptables-alerter-script\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402072 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/d0a39fe7-1d97-4306-97a7-0762a432e4bc-kube-api-access-v9z4l\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-cnibin\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-os-release\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-var-lib-kubelet\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqww\" (UniqueName: \"kubernetes.io/projected/ac401dbf-9a67-4224-884c-6a81c2e37a8e-kube-api-access-dpqww\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-cni-bin\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac401dbf-9a67-4224-884c-6a81c2e37a8e-host-slash\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-systemd\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-netns\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.403083 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-etc-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-cni-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-system-cni-dir\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysctl-d\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48345d08-5995-4c23-a63c-c55b32e8d7cd-host\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-socket-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ede2f80-924e-4425-92f0-6bc4ac661c93-tmp-dir\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-ovn\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-system-cni-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-conf-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f747861a-b056-485e-93f3-0d4491053813-tmp\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjdq\" (UniqueName: \"kubernetes.io/projected/6ede2f80-924e-4425-92f0-6bc4ac661c93-kube-api-access-vkjdq\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-kubelet\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-ovnkube-config\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-env-overrides\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.404123 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.402618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9b71ef0-3555-4870-9392-4955495398c0-kubelet-config\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.452757 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.452727 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 21:54:05 +0000 UTC" deadline="2028-03-12 17:54:59.905237465 +0000 UTC" Jul 07 21:59:06.452757 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.452752 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14731h55m53.452488097s" Jul 07 21:59:06.491486 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.491457 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 21:59:06.503791 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.503791 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-sys\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.503791 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48345d08-5995-4c23-a63c-c55b32e8d7cd-serviceca\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac401dbf-9a67-4224-884c-6a81c2e37a8e-iptables-alerter-script\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/d0a39fe7-1d97-4306-97a7-0762a432e4bc-kube-api-access-v9z4l\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-cnibin\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-os-release\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.503983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-var-lib-kubelet\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqww\" (UniqueName: \"kubernetes.io/projected/ac401dbf-9a67-4224-884c-6a81c2e37a8e-kube-api-access-dpqww\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504075 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-cnibin\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-cni-bin\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504110 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-cni-bin\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac401dbf-9a67-4224-884c-6a81c2e37a8e-host-slash\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-systemd\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac401dbf-9a67-4224-884c-6a81c2e37a8e-host-slash\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-netns\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504209 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-os-release\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-etc-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-cni-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504248 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-etc-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-system-cni-dir\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysctl-d\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504306 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-var-lib-kubelet\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48345d08-5995-4c23-a63c-c55b32e8d7cd-host\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-socket-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ede2f80-924e-4425-92f0-6bc4ac661c93-tmp-dir\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-ovn\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-system-cni-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-conf-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504491 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f747861a-b056-485e-93f3-0d4491053813-tmp\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjdq\" (UniqueName: \"kubernetes.io/projected/6ede2f80-924e-4425-92f0-6bc4ac661c93-kube-api-access-vkjdq\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-kubelet\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504513 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac401dbf-9a67-4224-884c-6a81c2e37a8e-iptables-alerter-script\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-ovnkube-config\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-env-overrides\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504572 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-systemd\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504513 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9b71ef0-3555-4870-9392-4955495398c0-kubelet-config\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-sys\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504640 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-kubelet\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504682 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-cni-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbw7b\" (UniqueName: \"kubernetes.io/projected/c0b7e5bc-a55f-4475-a9ce-a3f386594731-kube-api-access-dbw7b\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504741 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-run-ovn\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504772 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysctl-d\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504777 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-system-cni-dir\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.504968 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-sys-fs\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88226ee8-c447-4230-a118-614eb688c5f9-ovn-node-metrics-cert\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/402eead4-ad2d-4876-9e68-54be18236c77-cni-binary-copy\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504870 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-socket-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-modprobe-d\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504894 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48345d08-5995-4c23-a63c-c55b32e8d7cd-serviceca\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504932 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ede2f80-924e-4425-92f0-6bc4ac661c93-tmp-dir\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504980 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.504982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysconfig\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-lib-modules\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505097 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-k8s-cni-cncf-io\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-cni-multus\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505134 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-kubelet\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505144 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-netns\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k594k\" (UniqueName: \"kubernetes.io/projected/402eead4-ad2d-4876-9e68-54be18236c77-kube-api-access-k594k\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.505555 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505194 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-system-cni-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505236 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48345d08-5995-4c23-a63c-c55b32e8d7cd-host\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphv9\" (UniqueName: \"kubernetes.io/projected/f747861a-b056-485e-93f3-0d4491053813-kube-api-access-cphv9\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzv9\" (UniqueName: \"kubernetes.io/projected/48345d08-5995-4c23-a63c-c55b32e8d7cd-kube-api-access-rrzv9\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-systemd-units\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/402eead4-ad2d-4876-9e68-54be18236c77-multus-daemon-config\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cnibin\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-slash\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-socket-dir-parent\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-kubernetes\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-host\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-device-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-run-netns\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-var-lib-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-node-log\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-multus-certs\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-os-release\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.506523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505605 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/402eead4-ad2d-4876-9e68-54be18236c77-cni-binary-copy\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505623 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-lib-modules\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505652 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-kubelet\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505660 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-device-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505666 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-k8s-cni-cncf-io\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505706 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-node-log\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505711 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-run-netns\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505745 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysconfig\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505746 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-var-lib-cni-multus\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505786 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-cnibin\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505790 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-host-run-multus-certs\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505854 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-os-release\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505896 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-socket-dir-parent\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505907 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/402eead4-ad2d-4876-9e68-54be18236c77-multus-daemon-config\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysctl-conf\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.507301 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.505947 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-sysctl-conf\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506007 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f9b71ef0-3555-4870-9392-4955495398c0-kubelet-config\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506058 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-systemd-units\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506082 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-kubernetes\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506094 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-slash\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506128 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-multus-conf-dir\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.506156 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506167 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-host\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506216 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0b7e5bc-a55f-4475-a9ce-a3f386594731-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.506226 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:07.006207222 +0000 UTC m=+2.997780450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-registration-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-ovnkube-script-lib\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506306 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-env-overrides\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-hostroot\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506370 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-sys-fs\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-etc-kubernetes\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.508053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506407 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-var-lib-openvswitch\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506411 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-hostroot\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.506463 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506507 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-modprobe-d\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.506526 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:07.006509835 +0000 UTC m=+2.998083042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506536 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402eead4-ad2d-4876-9e68-54be18236c77-etc-kubernetes\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-run\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506578 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-registration-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f747861a-b056-485e-93f3-0d4491053813-etc-tuned\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506590 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-run\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a41a985-5a2d-425b-8723-2fffc40a1754-agent-certs\") pod \"konnectivity-agent-h8n52\" (UID: \"8a41a985-5a2d-425b-8723-2fffc40a1754\") " pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ede2f80-924e-4425-92f0-6bc4ac661c93-hosts-file\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-log-socket\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-cni-bin\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmf2\" (UniqueName: \"kubernetes.io/projected/88226ee8-c447-4230-a118-614eb688c5f9-kube-api-access-ftmf2\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506763 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-log-socket\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.508837 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9b71ef0-3555-4870-9392-4955495398c0-dbus\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506792 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-ovnkube-script-lib\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-systemd\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506862 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ede2f80-924e-4425-92f0-6bc4ac661c93-hosts-file\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506884 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88226ee8-c447-4230-a118-614eb688c5f9-ovnkube-config\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-cni-netd\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f747861a-b056-485e-93f3-0d4491053813-etc-systemd\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a41a985-5a2d-425b-8723-2fffc40a1754-konnectivity-ca\") pod \"konnectivity-agent-h8n52\" (UID: \"8a41a985-5a2d-425b-8723-2fffc40a1754\") " pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506938 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-cni-netd\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506940 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0b7e5bc-a55f-4475-a9ce-a3f386594731-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nspm\" (UniqueName: \"kubernetes.io/projected/17749b82-0e66-4e1d-a410-a6f414f06ff1-kube-api-access-4nspm\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506968 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d0a39fe7-1d97-4306-97a7-0762a432e4bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.506986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.507041 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88226ee8-c447-4230-a118-614eb688c5f9-host-cni-bin\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.509568 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.507144 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f9b71ef0-3555-4870-9392-4955495398c0-dbus\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:06.510176 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.507352 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a41a985-5a2d-425b-8723-2fffc40a1754-konnectivity-ca\") pod \"konnectivity-agent-h8n52\" (UID: \"8a41a985-5a2d-425b-8723-2fffc40a1754\") " pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.510176 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.508708 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f747861a-b056-485e-93f3-0d4491053813-tmp\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.510176 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.509619 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a41a985-5a2d-425b-8723-2fffc40a1754-agent-certs\") pod \"konnectivity-agent-h8n52\" (UID: \"8a41a985-5a2d-425b-8723-2fffc40a1754\") " pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.510318 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.510213 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f747861a-b056-485e-93f3-0d4491053813-etc-tuned\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.510380 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.510361 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88226ee8-c447-4230-a118-614eb688c5f9-ovn-node-metrics-cert\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.519748 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.517245 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z4l\" (UniqueName: \"kubernetes.io/projected/d0a39fe7-1d97-4306-97a7-0762a432e4bc-kube-api-access-v9z4l\") pod \"aws-ebs-csi-driver-node-rngh9\" (UID: \"d0a39fe7-1d97-4306-97a7-0762a432e4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.519748 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.518728 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjdq\" (UniqueName: \"kubernetes.io/projected/6ede2f80-924e-4425-92f0-6bc4ac661c93-kube-api-access-vkjdq\") pod \"node-resolver-7k7p6\" (UID: \"6ede2f80-924e-4425-92f0-6bc4ac661c93\") " pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.519748 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.519684 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k594k\" (UniqueName: \"kubernetes.io/projected/402eead4-ad2d-4876-9e68-54be18236c77-kube-api-access-k594k\") pod \"multus-w8gkx\" (UID: \"402eead4-ad2d-4876-9e68-54be18236c77\") " pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.520192 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.520169 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzv9\" (UniqueName: \"kubernetes.io/projected/48345d08-5995-4c23-a63c-c55b32e8d7cd-kube-api-access-rrzv9\") pod \"node-ca-8m9l6\" (UID: \"48345d08-5995-4c23-a63c-c55b32e8d7cd\") " pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:06.520282 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.520173 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmf2\" (UniqueName: \"kubernetes.io/projected/88226ee8-c447-4230-a118-614eb688c5f9-kube-api-access-ftmf2\") pod \"ovnkube-node-fm898\" (UID: \"88226ee8-c447-4230-a118-614eb688c5f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.520886 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.520861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqww\" (UniqueName: \"kubernetes.io/projected/ac401dbf-9a67-4224-884c-6a81c2e37a8e-kube-api-access-dpqww\") pod \"iptables-alerter-pgzth\" (UID: \"ac401dbf-9a67-4224-884c-6a81c2e37a8e\") " pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.521073 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.520869 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphv9\" (UniqueName: \"kubernetes.io/projected/f747861a-b056-485e-93f3-0d4491053813-kube-api-access-cphv9\") pod \"tuned-pkwtz\" (UID: \"f747861a-b056-485e-93f3-0d4491053813\") " pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.521220 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.521196 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nspm\" (UniqueName: \"kubernetes.io/projected/17749b82-0e66-4e1d-a410-a6f414f06ff1-kube-api-access-4nspm\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:06.522748 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.522724 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbw7b\" (UniqueName: \"kubernetes.io/projected/c0b7e5bc-a55f-4475-a9ce-a3f386594731-kube-api-access-dbw7b\") pod \"multus-additional-cni-plugins-jnfnp\" (UID: \"c0b7e5bc-a55f-4475-a9ce-a3f386594731\") " pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.607536 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.607491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:06.614906 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.614869 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:06.614906 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.614899 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:06.614906 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.614911 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:06.615159 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:06.614973 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 21:59:07.11495636 +0000 UTC m=+3.106529578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:06.699173 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.699133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" Jul 07 21:59:06.708775 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.708741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pgzth" Jul 07 21:59:06.722055 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.722002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" Jul 07 21:59:06.724993 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.724971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" Jul 07 21:59:06.731872 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.731841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7k7p6" Jul 07 21:59:06.740823 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.740790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:06.746934 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.746908 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:06.755886 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.755855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w8gkx" Jul 07 21:59:06.763754 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:06.763709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8m9l6" Jul 07 21:59:07.010406 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:07.010375 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:07.010597 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:07.010420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:07.010597 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.010554 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:07.010597 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.010566 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:07.010751 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.010621 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:08.010598645 +0000 UTC m=+4.002171852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:07.010751 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.010646 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:08.010627056 +0000 UTC m=+4.002200279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:07.211825 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:07.211735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:07.212010 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.211944 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:07.212010 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.211968 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:07.212010 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.211978 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:07.212151 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:07.212057 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 21:59:08.212042734 +0000 UTC m=+4.203615953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:07.452935 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:07.452864 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 21:54:05 +0000 UTC" deadline="2028-03-18 14:08:34.594171734 +0000 UTC" Jul 07 21:59:07.452935 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:07.452929 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14872h9m27.141245947s" Jul 07 21:59:07.578825 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.578646 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402eead4_ad2d_4876_9e68_54be18236c77.slice/crio-73249d269b24ef0ec6bd9fabe61082e8b3f28224971c316bb609da7398749637 WatchSource:0}: Error finding container 73249d269b24ef0ec6bd9fabe61082e8b3f28224971c316bb609da7398749637: Status 404 returned error can't find the container with id 73249d269b24ef0ec6bd9fabe61082e8b3f28224971c316bb609da7398749637 Jul 07 21:59:07.579577 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.579555 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88226ee8_c447_4230_a118_614eb688c5f9.slice/crio-b9c10ff4981c5dee2bab8135193e6e2a9f262f793ee48424f41064cab0f10bad WatchSource:0}: Error finding container b9c10ff4981c5dee2bab8135193e6e2a9f262f793ee48424f41064cab0f10bad: Status 404 returned error can't find the container with id b9c10ff4981c5dee2bab8135193e6e2a9f262f793ee48424f41064cab0f10bad Jul 07 21:59:07.586078 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.585671 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac401dbf_9a67_4224_884c_6a81c2e37a8e.slice/crio-3f7bc2b8774023b2636661f790c4464653755e2278efe4837c7d772bd8d98202 WatchSource:0}: Error finding container 3f7bc2b8774023b2636661f790c4464653755e2278efe4837c7d772bd8d98202: Status 404 returned error can't find the container with id 3f7bc2b8774023b2636661f790c4464653755e2278efe4837c7d772bd8d98202 Jul 07 21:59:07.586594 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.586573 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ede2f80_924e_4425_92f0_6bc4ac661c93.slice/crio-5a710cfa30eeeb8dae95c91e6e35d0825b58d7e80b2ef20c3f5dc801d6dfeac0 WatchSource:0}: Error finding container 5a710cfa30eeeb8dae95c91e6e35d0825b58d7e80b2ef20c3f5dc801d6dfeac0: Status 404 returned error can't find the container with id 5a710cfa30eeeb8dae95c91e6e35d0825b58d7e80b2ef20c3f5dc801d6dfeac0 Jul 07 21:59:07.587569 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.587300 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf747861a_b056_485e_93f3_0d4491053813.slice/crio-c27f30f02c265991e22cf9ec1627ffa841995ea5d6bc802bdc8f1797ac2f03d8 WatchSource:0}: Error finding container c27f30f02c265991e22cf9ec1627ffa841995ea5d6bc802bdc8f1797ac2f03d8: Status 404 returned error can't find the container with id c27f30f02c265991e22cf9ec1627ffa841995ea5d6bc802bdc8f1797ac2f03d8 Jul 07 21:59:07.589183 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.589087 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b7e5bc_a55f_4475_a9ce_a3f386594731.slice/crio-14317a1fc6ebc2f53eea74b6f961ff9ef70f642704f35575a3b526d9958cde23 WatchSource:0}: Error finding container 14317a1fc6ebc2f53eea74b6f961ff9ef70f642704f35575a3b526d9958cde23: Status 404 returned error can't find the container with id 14317a1fc6ebc2f53eea74b6f961ff9ef70f642704f35575a3b526d9958cde23 Jul 07 21:59:07.589998 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.589914 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48345d08_5995_4c23_a63c_c55b32e8d7cd.slice/crio-e3bd29603804827b3ec0113873651bb674e06da35167a6e2a24885d0d28b3a40 WatchSource:0}: Error finding container e3bd29603804827b3ec0113873651bb674e06da35167a6e2a24885d0d28b3a40: Status 404 returned error can't find the container with id e3bd29603804827b3ec0113873651bb674e06da35167a6e2a24885d0d28b3a40 Jul 07 21:59:07.593054 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:07.592872 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a41a985_5a2d_425b_8723_2fffc40a1754.slice/crio-7a3379296759f712a5cb883899c6758de455538c704aaad6d863d38c924e630e WatchSource:0}: Error finding container 7a3379296759f712a5cb883899c6758de455538c704aaad6d863d38c924e630e: Status 404 returned error can't find the container with id 7a3379296759f712a5cb883899c6758de455538c704aaad6d863d38c924e630e Jul 07 21:59:08.018675 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.017868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:08.018675 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.017970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:08.018675 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.018116 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:08.018675 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.018181 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:10.018160466 +0000 UTC m=+6.009733695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:08.018675 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.018571 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:08.018675 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.018633 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:10.018617206 +0000 UTC m=+6.010190423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:08.220733 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.219993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:08.220733 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.220231 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:08.220733 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.220258 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:08.220733 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.220272 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:08.220733 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.220333 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 21:59:10.220314719 +0000 UTC m=+6.211887926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:08.573049 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.572949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:08.573465 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.573101 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:08.573543 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.573495 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:08.573634 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.573607 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:08.573703 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.573682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:08.573767 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:08.573752 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:08.590881 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.590840 2565 generic.go:356] "Generic (PLEG): container finished" podID="223e82db7a0ce00bd93fd50f651c4d28" containerID="77eb7ca1a326ff15a150979e8a9ec5318784c178ed41e54bebb858e012d1710e" exitCode=0 Jul 07 21:59:08.591096 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.590966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" event={"ID":"223e82db7a0ce00bd93fd50f651c4d28","Type":"ContainerDied","Data":"77eb7ca1a326ff15a150979e8a9ec5318784c178ed41e54bebb858e012d1710e"} Jul 07 21:59:08.615112 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.612496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" event={"ID":"b5456099913828cc9faa58b08483592f","Type":"ContainerStarted","Data":"eeffe25e1575e523651407cb7e9dcd5b9040bbaf57a983b96ed5153691c7a5ae"} Jul 07 21:59:08.621645 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.621610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8n52" event={"ID":"8a41a985-5a2d-425b-8723-2fffc40a1754","Type":"ContainerStarted","Data":"7a3379296759f712a5cb883899c6758de455538c704aaad6d863d38c924e630e"} Jul 07 21:59:08.627793 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.627684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8m9l6" event={"ID":"48345d08-5995-4c23-a63c-c55b32e8d7cd","Type":"ContainerStarted","Data":"e3bd29603804827b3ec0113873651bb674e06da35167a6e2a24885d0d28b3a40"} Jul 07 21:59:08.635062 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.634918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" event={"ID":"f747861a-b056-485e-93f3-0d4491053813","Type":"ContainerStarted","Data":"c27f30f02c265991e22cf9ec1627ffa841995ea5d6bc802bdc8f1797ac2f03d8"} Jul 07 21:59:08.655523 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.655411 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pgzth" event={"ID":"ac401dbf-9a67-4224-884c-6a81c2e37a8e","Type":"ContainerStarted","Data":"3f7bc2b8774023b2636661f790c4464653755e2278efe4837c7d772bd8d98202"} Jul 07 21:59:08.680166 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.680108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w8gkx" event={"ID":"402eead4-ad2d-4876-9e68-54be18236c77","Type":"ContainerStarted","Data":"73249d269b24ef0ec6bd9fabe61082e8b3f28224971c316bb609da7398749637"} Jul 07 21:59:08.694991 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.694951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerStarted","Data":"14317a1fc6ebc2f53eea74b6f961ff9ef70f642704f35575a3b526d9958cde23"} Jul 07 21:59:08.717081 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.716805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7k7p6" event={"ID":"6ede2f80-924e-4425-92f0-6bc4ac661c93","Type":"ContainerStarted","Data":"5a710cfa30eeeb8dae95c91e6e35d0825b58d7e80b2ef20c3f5dc801d6dfeac0"} Jul 07 21:59:08.724646 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.724521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"b9c10ff4981c5dee2bab8135193e6e2a9f262f793ee48424f41064cab0f10bad"} Jul 07 21:59:08.736303 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:08.736262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" event={"ID":"d0a39fe7-1d97-4306-97a7-0762a432e4bc","Type":"ContainerStarted","Data":"39fbdecb6bbfe7cac44a44e182e2d955ad7bed5ac1e908eab2b5651b91af181c"} Jul 07 21:59:09.751139 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:09.751078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" event={"ID":"223e82db7a0ce00bd93fd50f651c4d28","Type":"ContainerStarted","Data":"34c249dc5725d141670205254fdf3e7bae2e7fd0a348f675ee54f2fe252a331a"} Jul 07 21:59:09.767602 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:09.766976 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-137.ec2.internal" podStartSLOduration=4.766948686 podStartE2EDuration="4.766948686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:08.634047189 +0000 UTC m=+4.625620413" watchObservedRunningTime="2026-07-07 21:59:09.766948686 +0000 UTC m=+5.758521916" Jul 07 21:59:10.040064 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:10.039926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:10.040245 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:10.040092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:10.040245 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.040113 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:10.040245 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.040194 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:14.040173086 +0000 UTC m=+10.031746300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:10.040245 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.040216 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:10.040459 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.040274 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:14.040255146 +0000 UTC m=+10.031828368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:10.242808 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:10.242767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:10.243117 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.242944 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:10.243117 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.242968 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:10.243117 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.242982 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:10.243117 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.243061 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 21:59:14.243042375 +0000 UTC m=+10.234615586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:10.572541 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:10.572502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:10.572716 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.572658 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:10.573124 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:10.573093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:10.573276 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.573236 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:10.573341 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:10.573310 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:10.573410 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:10.573388 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:12.570308 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:12.570271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:12.570720 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:12.570409 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:12.570821 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:12.570801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:12.570980 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:12.570911 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:12.571091 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:12.571076 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:12.571175 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:12.571156 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:14.077531 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:14.077487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:14.078007 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:14.077557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:14.078007 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.077724 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:14.078007 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.077789 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:22.077769226 +0000 UTC m=+18.069342447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:14.078007 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.077864 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:14.078007 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.077898 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:22.077885527 +0000 UTC m=+18.069458747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:14.279952 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:14.279314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:14.279952 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.279520 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:14.279952 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.279541 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:14.279952 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.279554 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:14.279952 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.279615 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 21:59:22.279596959 +0000 UTC m=+18.271170167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:14.572260 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:14.569929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:14.572260 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.570075 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:14.572260 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:14.571838 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:14.572260 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.571948 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:14.572260 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:14.572178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:14.572540 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:14.572293 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:16.572431 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:16.572329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:16.572891 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:16.572332 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:16.572891 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:16.572466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:16.572891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:16.572460 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:16.572891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:16.572559 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:16.572891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:16.572639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:18.570242 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:18.570206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:18.570722 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:18.570262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:18.570722 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:18.570210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:18.570722 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:18.570360 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:18.570722 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:18.570499 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:18.570722 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:18.570596 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:20.570390 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:20.570351 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:20.570891 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:20.570350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:20.570891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:20.570498 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:20.570891 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:20.570362 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:20.570891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:20.570594 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:20.570891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:20.570650 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:22.143770 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:22.143730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:22.144322 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:22.143786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:22.144322 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.143906 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:22.144322 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.143943 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:22.144322 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.143989 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:38.143968616 +0000 UTC m=+34.135541848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:22.144322 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.144010 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:38.144002869 +0000 UTC m=+34.135576075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:22.345199 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:22.345157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:22.345377 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.345354 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:22.345423 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.345383 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:22.345423 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.345397 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:22.345520 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.345465 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 21:59:38.345445471 +0000 UTC m=+34.337018701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:22.570885 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:22.570795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:22.570885 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:22.570842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:22.571129 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:22.570803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:22.571129 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.570945 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:22.571129 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.571039 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:22.571270 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:22.571132 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:24.572040 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:24.571533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:24.572040 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:24.571650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:24.572040 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:24.571661 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:24.572040 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:24.571742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:24.572040 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:24.571754 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:24.572040 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:24.571822 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:26.570510 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:26.570464 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:26.571053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:26.570533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:26.571053 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:26.570532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:26.571053 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:26.570616 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:26.571053 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:26.570671 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:26.571053 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:26.570762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:27.791815 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.791343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8n52" event={"ID":"8a41a985-5a2d-425b-8723-2fffc40a1754","Type":"ContainerStarted","Data":"880aadc5d52d65266b76a7ffd8899d63c017f30b9559811c66e8c118c9653369"} Jul 07 21:59:27.793672 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.793639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8m9l6" event={"ID":"48345d08-5995-4c23-a63c-c55b32e8d7cd","Type":"ContainerStarted","Data":"dbe81e6663c213ba5c508fc192234475f0a56b67be98d340e7b1db9d588c3df8"} Jul 07 21:59:27.795188 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.795159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" event={"ID":"f747861a-b056-485e-93f3-0d4491053813","Type":"ContainerStarted","Data":"fb070c24bb2385f2d08cd5cb5c3e234113036b2a8c0397ba2dce5ea0b01dd7de"} Jul 07 21:59:27.796711 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.796686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w8gkx" event={"ID":"402eead4-ad2d-4876-9e68-54be18236c77","Type":"ContainerStarted","Data":"9450191385da369574818fb73da32d22c7460d3feb10f78380b391678b2f2c27"} Jul 07 21:59:27.798364 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.798337 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b7e5bc-a55f-4475-a9ce-a3f386594731" containerID="dd4a5224e6099ec46043b07de240d9162c3542ab812d683e9410dd00ecdfdd75" exitCode=0 Jul 07 21:59:27.798481 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.798414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerDied","Data":"dd4a5224e6099ec46043b07de240d9162c3542ab812d683e9410dd00ecdfdd75"} Jul 07 21:59:27.800074 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.800051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7k7p6" event={"ID":"6ede2f80-924e-4425-92f0-6bc4ac661c93","Type":"ContainerStarted","Data":"46a53040aacfbaf9c1cf7910d0732a9c3adf04e0b2a11a46fd68831a3b62b902"} Jul 07 21:59:27.803584 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.803558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"8cc1503e0174191a95773d96c30563590d192b841c10676ccc19feb80398b365"} Jul 07 21:59:27.803709 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.803635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"1e45eb902398859bd96acd0d2386a289896fff3b2e00ba477c446d5c23a3d9c8"} Jul 07 21:59:27.803709 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.803688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"e240ace31d1ce38fb6e44c14db4c5336f85af07c2c2c18c2f07591585fa031df"} Jul 07 21:59:27.803820 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.803783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"4f0d3a583b4f064dac95276ca7de9d00e049bcdd56fb07ff1439ebba6a09a28c"} Jul 07 21:59:27.803875 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.803826 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"e8c37dca862c25106580bb01702c9d232b59c262e8c93e021fa40f6f6781b0af"} Jul 07 21:59:27.803875 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.803860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"6ab5083ccef353fa39287e71bcb783cdfe26c0a1cd8a22236e040a5ccbdd7fc8"} Jul 07 21:59:27.805154 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.805125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" event={"ID":"d0a39fe7-1d97-4306-97a7-0762a432e4bc","Type":"ContainerStarted","Data":"a22bc90f52d79aca88678f2b6a0f6135994320c21ab27a5628a5a71704d37f1f"} Jul 07 21:59:27.807291 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.807242 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-137.ec2.internal" podStartSLOduration=22.807223891 podStartE2EDuration="22.807223891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:09.76794594 +0000 UTC m=+5.759519169" watchObservedRunningTime="2026-07-07 21:59:27.807223891 +0000 UTC m=+23.798797120" Jul 07 21:59:27.807572 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.807545 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h8n52" podStartSLOduration=8.359417173 podStartE2EDuration="23.807534812s" totalImagesPullingTime="15.448117639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:27.806819173 +0000 UTC m=+23.798392401" watchObservedRunningTime="2026-07-07 21:59:27.807534812 +0000 UTC m=+23.799108041" Jul 07 21:59:27.824626 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.824574 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pkwtz" podStartSLOduration=4.521369984 podStartE2EDuration="23.824554291s" totalImagesPullingTime="19.303184307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:27.824045907 +0000 UTC m=+23.815619135" watchObservedRunningTime="2026-07-07 21:59:27.824554291 +0000 UTC m=+23.816127518" Jul 07 21:59:27.838815 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.838749 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8m9l6" podStartSLOduration=4.550567609 podStartE2EDuration="23.838725726s" totalImagesPullingTime="19.288158117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:27.837820671 +0000 UTC m=+23.829393899" watchObservedRunningTime="2026-07-07 21:59:27.838725726 +0000 UTC m=+23.830298954" Jul 07 21:59:27.852685 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.852636 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7k7p6" podStartSLOduration=4.557018257 podStartE2EDuration="23.852620343s" totalImagesPullingTime="19.295602086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:27.852146143 +0000 UTC m=+23.843719372" watchObservedRunningTime="2026-07-07 21:59:27.852620343 +0000 UTC m=+23.844193572" Jul 07 21:59:27.893672 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.893626 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w8gkx" podStartSLOduration=4.533892202 podStartE2EDuration="23.893608857s" totalImagesPullingTime="19.359716655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:27.8723938 +0000 UTC m=+23.863967029" watchObservedRunningTime="2026-07-07 21:59:27.893608857 +0000 UTC m=+23.885182086" Jul 07 21:59:27.944219 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:27.944185 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:28.011058 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.011010 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:28.078419 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.078336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:28.320552 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.320513 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 21:59:28.470580 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.470477 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T21:59:28.320544398Z","UUID":"89884ea8-9137-4e28-92d8-f41c2eca84af","Handler":null,"Name":"","Endpoint":""} Jul 07 21:59:28.472185 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.472161 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 21:59:28.472317 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.472203 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 21:59:28.567126 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.567096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:28.570943 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.570913 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:28.571100 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.570926 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:28.571100 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.571057 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:28.571100 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:28.571062 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:28.571229 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:28.571148 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:28.571272 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:28.571231 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:28.663388 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.663312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:28.737638 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.737593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:28.809816 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.809779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pgzth" event={"ID":"ac401dbf-9a67-4224-884c-6a81c2e37a8e","Type":"ContainerStarted","Data":"2c5dfdcb1e10f5254aa9439de7a442680194df4c2862837214ba4ef0d16c64f3"} Jul 07 21:59:28.813056 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.813014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" event={"ID":"d0a39fe7-1d97-4306-97a7-0762a432e4bc","Type":"ContainerStarted","Data":"d1b8ab1cf879c7df961a4ab8c2a9b74b0eac8a1731042d8da16d5bade794adb9"} Jul 07 21:59:28.835927 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.835878 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pgzth" podStartSLOduration=5.548432663 podStartE2EDuration="24.835863776s" totalImagesPullingTime="19.287431113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:28.83419799 +0000 UTC m=+24.825771218" watchObservedRunningTime="2026-07-07 21:59:28.835863776 +0000 UTC m=+24.827437003" Jul 07 21:59:28.878145 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:28.878110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:29.109296 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.109217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:29.236892 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.236856 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:29.337888 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.337850 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:29.723109 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.723079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:29.799654 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.799569 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:29.819426 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.819389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"2a96fd75049b1d0f8111e42825cb17845b48446a29ba423e4a0fb9f8157d4cd2"} Jul 07 21:59:29.821740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.821705 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" event={"ID":"d0a39fe7-1d97-4306-97a7-0762a432e4bc","Type":"ContainerStarted","Data":"3fe637c96f5aea184afebf1aafa35781ee0e82971f666115c3d9653b8ab678ed"} Jul 07 21:59:29.842871 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:29.842812 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rngh9" podStartSLOduration=4.267889389 podStartE2EDuration="25.842790099s" totalImagesPullingTime="21.57490071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:29.841840934 +0000 UTC m=+25.833414162" watchObservedRunningTime="2026-07-07 21:59:29.842790099 +0000 UTC m=+25.834363328" Jul 07 21:59:30.001406 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:30.001084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:30.107081 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:30.106972 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:30.574495 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:30.574409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:30.574676 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:30.574516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:30.574676 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:30.574530 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:30.574676 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:30.574523 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:30.574676 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:30.574666 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:30.574894 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:30.574758 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:31.028253 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:31.028177 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7k7p6_6ede2f80-924e-4425-92f0-6bc4ac661c93/dns-node-resolver/0.log" Jul 07 21:59:31.281443 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:31.281358 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:31.372702 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:31.372665 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:31.691139 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:31.691093 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:31.792349 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:31.792313 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:32.028394 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.028314 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8m9l6_48345d08-5995-4c23-a63c-c55b32e8d7cd/node-ca/0.log" Jul 07 21:59:32.040480 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.040451 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:32.041099 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.041080 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:32.570398 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.570154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:32.570562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.570154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:32.570562 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:32.570411 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:32.570562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.570221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:32.570562 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:32.570504 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:32.570722 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:32.570583 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:32.793753 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.793712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:32.833120 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.833003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" event={"ID":"88226ee8-c447-4230-a118-614eb688c5f9","Type":"ContainerStarted","Data":"14a11d0e98449a12501dc7d4a8af00440d58ec3e3ed0e4e20d5442570dd400f4"} Jul 07 21:59:32.833288 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.833238 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:32.835012 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.834986 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b7e5bc-a55f-4475-a9ce-a3f386594731" containerID="11f08c50896623f6dbe9bfdba6f2edd2ed5f096f03bf1529f07f716dca966167" exitCode=0 Jul 07 21:59:32.835156 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.835074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerDied","Data":"11f08c50896623f6dbe9bfdba6f2edd2ed5f096f03bf1529f07f716dca966167"} Jul 07 21:59:32.850515 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.850488 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:32.926475 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:32.926418 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" podStartSLOduration=9.560322921000001 podStartE2EDuration="28.92640171s" totalImagesPullingTime="19.366078789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:32.883100236 +0000 UTC m=+28.874673464" watchObservedRunningTime="2026-07-07 21:59:32.92640171 +0000 UTC m=+28.917974938" Jul 07 21:59:33.372664 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:33.372625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:33.839002 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:33.838907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:33.839212 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:33.839063 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:33.854298 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:33.854268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 21:59:34.283806 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.283774 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zzjj"] Jul 07 21:59:34.284044 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.283919 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:34.284101 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:34.284043 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:34.287353 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.287322 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qf59h"] Jul 07 21:59:34.287488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.287455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:34.287586 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:34.287561 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:34.287926 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.287907 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cz4t2"] Jul 07 21:59:34.288041 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.288012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:34.288142 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:34.288124 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:34.354238 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.354147 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:34.354383 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.354303 2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 21:59:34.354861 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.354846 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h8n52" Jul 07 21:59:34.842810 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.842714 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b7e5bc-a55f-4475-a9ce-a3f386594731" containerID="4d1cc35a46be35b48bbb65467adbd051f80535cda3b6d46f181e4df4b91d235d" exitCode=0 Jul 07 21:59:34.842810 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:34.842749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerDied","Data":"4d1cc35a46be35b48bbb65467adbd051f80535cda3b6d46f181e4df4b91d235d"} Jul 07 21:59:35.690720 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:35.690537 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:36.111441 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.111404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:36.212689 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.212608 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:36.314394 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.314357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:36.413660 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.413625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:36.552741 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.552709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:36.570585 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.570559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:36.570740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.570562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:36.570740 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:36.570681 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:36.570740 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.570562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:36.570873 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:36.570737 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:36.570873 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:36.570836 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:36.850472 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.850386 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b7e5bc-a55f-4475-a9ce-a3f386594731" containerID="9275c8625f9d92526aad8e15921fee846bede99f1e7a95cc383be815e8157505" exitCode=0 Jul 07 21:59:36.850472 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.850443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerDied","Data":"9275c8625f9d92526aad8e15921fee846bede99f1e7a95cc383be815e8157505"} Jul 07 21:59:36.872562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:36.872534 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:37.112317 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:37.112225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:37.228237 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:37.228199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:37.807418 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:37.807374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:38.165818 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.165665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:38.165818 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.165725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:38.166294 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.165846 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:38.166294 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.165866 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:38.166294 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.165939 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs podName:17749b82-0e66-4e1d-a410-a6f414f06ff1 nodeName:}" failed. No retries permitted until 2026-07-07 22:00:10.165909037 +0000 UTC m=+66.157482263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs") pod "network-metrics-daemon-4zzjj" (UID: "17749b82-0e66-4e1d-a410-a6f414f06ff1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 21:59:38.166294 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.165992 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret podName:f9b71ef0-3555-4870-9392-4955495398c0 nodeName:}" failed. No retries permitted until 2026-07-07 22:00:10.1659737 +0000 UTC m=+66.157546924 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret") pod "global-pull-secret-syncer-cz4t2" (UID: "f9b71ef0-3555-4870-9392-4955495398c0") : object "kube-system"/"original-pull-secret" not registered Jul 07 21:59:38.213638 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.213598 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:38.368161 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.368123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:38.368324 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.368280 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 21:59:38.368324 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.368302 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 21:59:38.368324 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.368315 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gq58x for pod openshift-network-diagnostics/network-check-target-qf59h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:38.368464 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.368385 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x podName:f64c2e17-a818-4c03-a33b-44817259494e nodeName:}" failed. No retries permitted until 2026-07-07 22:00:10.368367079 +0000 UTC m=+66.359940306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq58x" (UniqueName: "kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x") pod "network-check-target-qf59h" (UID: "f64c2e17-a818-4c03-a33b-44817259494e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 21:59:38.405266 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.405227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:38.570539 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.570450 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:38.570709 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.570583 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:38.570709 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.570609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:38.570812 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.570752 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:38.570812 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.570465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:38.570893 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:38.570855 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:38.608498 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.608458 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:38.800221 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:38.800182 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:39.127315 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:39.127276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:39.607932 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:39.607846 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:39.824492 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:39.824458 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:39.927354 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:39.927267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:40.232142 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.232044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:40.422924 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.422889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:40.570387 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.570309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:40.570554 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.570309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:40.570554 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:40.570443 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:40.570554 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.570309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:40.570554 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:40.570523 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:40.570752 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:40.570587 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:40.706901 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.706862 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:40.806706 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:40.806674 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:41.010925 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:41.010886 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:41.411773 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:41.411718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:41.865166 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:41.865085 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b7e5bc-a55f-4475-a9ce-a3f386594731" containerID="c795d71ef7501f513c603c4d172c23eeccfb25b778becf0ff0b36a21a9a8c9c2" exitCode=0 Jul 07 21:59:41.865166 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:41.865142 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerDied","Data":"c795d71ef7501f513c603c4d172c23eeccfb25b778becf0ff0b36a21a9a8c9c2"} Jul 07 21:59:42.009671 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.009636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:42.083497 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.083425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:42.571113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.571075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:42.571113 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.571093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:42.571363 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.571179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:42.571363 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:42.571185 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:42.571363 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:42.571276 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:42.571488 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:42.571377 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:42.604422 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.604386 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:42.824963 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.824874 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:42.871135 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.871105 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0b7e5bc-a55f-4475-a9ce-a3f386594731" containerID="40b941bb4f6a3a714daf697f4d01955141f6b464b880be043691c065566ebe04" exitCode=0 Jul 07 21:59:42.871497 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:42.871155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerDied","Data":"40b941bb4f6a3a714daf697f4d01955141f6b464b880be043691c065566ebe04"} Jul 07 21:59:43.007090 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:43.006900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:43.402556 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:43.402518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:43.877072 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:43.877015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" event={"ID":"c0b7e5bc-a55f-4475-a9ce-a3f386594731","Type":"ContainerStarted","Data":"c9448b140bfac9895c801936073b172c26e4188ae227cdfdd5e9d0c04780a312"} Jul 07 21:59:43.899738 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:43.899684 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jnfnp" podStartSLOduration=10.14157438 podStartE2EDuration="39.899665852s" totalImagesPullingTime="29.758091472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 21:59:43.898528817 +0000 UTC m=+39.890102046" watchObservedRunningTime="2026-07-07 21:59:43.899665852 +0000 UTC m=+39.891239079" Jul 07 21:59:44.000982 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.000949 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:44.323479 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.323404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:44.428735 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.428687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:44.525230 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.525192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:44.571234 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.571199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:44.571404 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.571277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:44.571404 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:44.571294 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:44.571404 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:44.571388 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:44.571502 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.571444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:44.571530 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:44.571500 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:44.625163 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.625087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:44.861214 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.861166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:44.961471 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:44.961390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:45.067522 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:45.067494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:45.600857 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:45.600813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.003469 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.003381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.247040 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.246993 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.349870 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.349790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.448981 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.448948 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.539661 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.539623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.570660 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.570625 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:46.570660 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.570657 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:46.570891 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.570630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:46.570891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:46.570741 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:46.570891 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:46.570828 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:46.571064 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:46.570901 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:46.613978 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.613887 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.714912 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.714876 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:46.816012 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:46.815976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:47.293011 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:47.292980 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:47.508299 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:47.508266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:47.609147 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:47.609062 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:47.896347 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:47.896266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:48.067428 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:48.067397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:48.167792 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:48.167702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:48.570722 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:48.570686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:48.571161 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:48.570733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:48.571161 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:48.570698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:48.571161 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:48.570826 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:48.571161 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:48.570897 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:48.571161 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:48.570971 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:50.570688 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:50.570649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:50.570688 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:50.570673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:50.571195 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:50.570649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:50.571195 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:50.570759 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:50.571195 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:50.570849 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:50.571195 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:50.570946 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:52.094915 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:52.094875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:52.158255 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:52.158216 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:52.570423 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:52.570375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:52.570607 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:52.570375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:52.570607 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:52.570518 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:52.570607 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:52.570563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:52.570742 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:52.570672 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:52.570783 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:52.570740 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:53.769044 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:53.769001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:53.886798 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:53.886756 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:54.079130 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.079039 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:54.178142 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.178111 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:54.273788 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.273755 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:54.571719 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.571687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:54.571914 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:54.571777 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cz4t2" podUID="f9b71ef0-3555-4870-9392-4955495398c0" Jul 07 21:59:54.571914 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.571857 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:54.572053 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:54.571961 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zzjj" podUID="17749b82-0e66-4e1d-a410-a6f414f06ff1" Jul 07 21:59:54.572118 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.572048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:54.572176 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:54.572151 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf59h" podUID="f64c2e17-a818-4c03-a33b-44817259494e" Jul 07 21:59:54.769480 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.769452 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:54.839829 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.839797 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-137.ec2.internal" event="NodeReady" Jul 07 21:59:54.839997 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.839966 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 21:59:54.864111 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.864085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:54.900901 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.900867 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-46cgj"] Jul 07 21:59:54.920811 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.920769 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b9q6n"] Jul 07 21:59:54.921018 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.920920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:54.924339 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.924302 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6c5lw\"" Jul 07 21:59:54.924486 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.924343 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 21:59:54.924853 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.924814 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 21:59:54.934794 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.934763 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9q6n"] Jul 07 21:59:54.934919 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.934807 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46cgj"] Jul 07 21:59:54.934919 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.934887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:54.937733 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.937707 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 21:59:54.937733 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.937730 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 21:59:54.937924 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.937798 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 21:59:54.937924 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.937862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 21:59:54.938075 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:54.938065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lz8kg\"" Jul 07 21:59:55.003164 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.003129 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4s4fw"] Jul 07 21:59:55.020817 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.020751 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4s4fw"] Jul 07 21:59:55.020946 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.020863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.023721 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.023692 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 21:59:55.023848 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.023752 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 21:59:55.024070 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.024051 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 21:59:55.024199 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.024118 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-czs54\"" Jul 07 21:59:55.066370 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.066331 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:55.090389 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-tmp-dir\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.090389 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrg5\" (UniqueName: \"kubernetes.io/projected/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-kube-api-access-kwrg5\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.090563 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8673fd68-bfb5-4761-950b-bdc1d1ac5020-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.090563 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8673fd68-bfb5-4761-950b-bdc1d1ac5020-crio-socket\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.090563 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8673fd68-bfb5-4761-950b-bdc1d1ac5020-data-volume\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.090690 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-config-volume\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.090690 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-metrics-tls\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.090690 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8673fd68-bfb5-4761-950b-bdc1d1ac5020-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.090690 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.090660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5nl\" (UniqueName: \"kubernetes.io/projected/8673fd68-bfb5-4761-950b-bdc1d1ac5020-kube-api-access-vp5nl\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.140866 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.140833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:55.191593 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c673e5-2c40-47bb-92d5-0d73da843886-cert\") pod \"ingress-canary-4s4fw\" (UID: \"71c673e5-2c40-47bb-92d5-0d73da843886\") " pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.191753 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-config-volume\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.191753 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-metrics-tls\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.191753 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8673fd68-bfb5-4761-950b-bdc1d1ac5020-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.191753 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5nl\" (UniqueName: \"kubernetes.io/projected/8673fd68-bfb5-4761-950b-bdc1d1ac5020-kube-api-access-vp5nl\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.191957 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-tmp-dir\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.191957 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrg5\" (UniqueName: \"kubernetes.io/projected/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-kube-api-access-kwrg5\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.191957 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8673fd68-bfb5-4761-950b-bdc1d1ac5020-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.191957 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8673fd68-bfb5-4761-950b-bdc1d1ac5020-crio-socket\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.191957 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8673fd68-bfb5-4761-950b-bdc1d1ac5020-data-volume\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.191957 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.191903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw6f\" (UniqueName: \"kubernetes.io/projected/71c673e5-2c40-47bb-92d5-0d73da843886-kube-api-access-cfw6f\") pod \"ingress-canary-4s4fw\" (UID: \"71c673e5-2c40-47bb-92d5-0d73da843886\") " pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.192272 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.192132 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8673fd68-bfb5-4761-950b-bdc1d1ac5020-crio-socket\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.192322 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.192268 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8673fd68-bfb5-4761-950b-bdc1d1ac5020-data-volume\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.192322 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.192307 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-tmp-dir\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.192508 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.192478 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8673fd68-bfb5-4761-950b-bdc1d1ac5020-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.192578 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.192556 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-config-volume\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.195970 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.195946 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-metrics-tls\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.196097 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.195946 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8673fd68-bfb5-4761-950b-bdc1d1ac5020-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.201238 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.201214 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5nl\" (UniqueName: \"kubernetes.io/projected/8673fd68-bfb5-4761-950b-bdc1d1ac5020-kube-api-access-vp5nl\") pod \"insights-runtime-extractor-b9q6n\" (UID: \"8673fd68-bfb5-4761-950b-bdc1d1ac5020\") " pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.201430 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.201410 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrg5\" (UniqueName: \"kubernetes.io/projected/62047d1c-5895-4bda-b8a0-3a57fbc5eeb9-kube-api-access-kwrg5\") pod \"dns-default-46cgj\" (UID: \"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9\") " pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.232384 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.232349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46cgj" Jul 07 21:59:55.243549 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.243520 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9q6n" Jul 07 21:59:55.292994 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.292963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw6f\" (UniqueName: \"kubernetes.io/projected/71c673e5-2c40-47bb-92d5-0d73da843886-kube-api-access-cfw6f\") pod \"ingress-canary-4s4fw\" (UID: \"71c673e5-2c40-47bb-92d5-0d73da843886\") " pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.292994 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.292996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c673e5-2c40-47bb-92d5-0d73da843886-cert\") pod \"ingress-canary-4s4fw\" (UID: \"71c673e5-2c40-47bb-92d5-0d73da843886\") " pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.295786 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.295757 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71c673e5-2c40-47bb-92d5-0d73da843886-cert\") pod \"ingress-canary-4s4fw\" (UID: \"71c673e5-2c40-47bb-92d5-0d73da843886\") " pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.302372 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.302340 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw6f\" (UniqueName: \"kubernetes.io/projected/71c673e5-2c40-47bb-92d5-0d73da843886-kube-api-access-cfw6f\") pod \"ingress-canary-4s4fw\" (UID: \"71c673e5-2c40-47bb-92d5-0d73da843886\") " pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.330274 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.330233 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4s4fw" Jul 07 21:59:55.441543 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.441504 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46cgj"] Jul 07 21:59:55.444968 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:55.444939 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62047d1c_5895_4bda_b8a0_3a57fbc5eeb9.slice/crio-57649ea805d911e60eb0fb079e82dbd612757afd485a8beda7a718826ec1be3c WatchSource:0}: Error finding container 57649ea805d911e60eb0fb079e82dbd612757afd485a8beda7a718826ec1be3c: Status 404 returned error can't find the container with id 57649ea805d911e60eb0fb079e82dbd612757afd485a8beda7a718826ec1be3c Jul 07 21:59:55.459217 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.459187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9q6n"] Jul 07 21:59:55.463218 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:55.463180 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8673fd68_bfb5_4761_950b_bdc1d1ac5020.slice/crio-ffac468a3b6132926332372e5781c4a671a114d72f40b532f21b4bc49a9abdb1 WatchSource:0}: Error finding container ffac468a3b6132926332372e5781c4a671a114d72f40b532f21b4bc49a9abdb1: Status 404 returned error can't find the container with id ffac468a3b6132926332372e5781c4a671a114d72f40b532f21b4bc49a9abdb1 Jul 07 21:59:55.502154 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.502115 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4s4fw"] Jul 07 21:59:55.505433 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:55.505405 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c673e5_2c40_47bb_92d5_0d73da843886.slice/crio-e311056d97ae8393ce35ba012e77584e36a44ed996a1fd26e6f323458088cee5 WatchSource:0}: Error finding container e311056d97ae8393ce35ba012e77584e36a44ed996a1fd26e6f323458088cee5: Status 404 returned error can't find the container with id e311056d97ae8393ce35ba012e77584e36a44ed996a1fd26e6f323458088cee5 Jul 07 21:59:55.906992 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.906950 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9q6n" event={"ID":"8673fd68-bfb5-4761-950b-bdc1d1ac5020","Type":"ContainerStarted","Data":"ffac468a3b6132926332372e5781c4a671a114d72f40b532f21b4bc49a9abdb1"} Jul 07 21:59:55.908372 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.908342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46cgj" event={"ID":"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9","Type":"ContainerStarted","Data":"57649ea805d911e60eb0fb079e82dbd612757afd485a8beda7a718826ec1be3c"} Jul 07 21:59:55.909744 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:55.909715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4s4fw" event={"ID":"71c673e5-2c40-47bb-92d5-0d73da843886","Type":"ContainerStarted","Data":"e311056d97ae8393ce35ba012e77584e36a44ed996a1fd26e6f323458088cee5"} Jul 07 21:59:56.570662 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.570157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 21:59:56.570662 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.570016 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 21:59:56.571816 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.571061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 21:59:56.573285 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.573114 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 21:59:56.574827 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.574513 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 21:59:56.574827 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.574730 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqsmx\"" Jul 07 21:59:56.574990 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.574876 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 21:59:56.574990 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.574982 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 21:59:56.575116 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.575000 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8gff\"" Jul 07 21:59:56.726271 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.726221 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8f7d455bf-fmsbb"] Jul 07 21:59:56.742204 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.741474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:56.742204 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.741647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f7d455bf-fmsbb"] Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.743734 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.743977 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.744336 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.744520 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.744688 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.744867 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-65fgl\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.745154 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 21:59:56.745280 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.745246 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 21:59:56.905488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.905075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-oauth-config\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:56.905488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.905124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-config\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:56.905488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.905182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-service-ca\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:56.905488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.905251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqvs\" (UniqueName: \"kubernetes.io/projected/54a6416d-fadc-4fd9-80d4-51a89956dca3-kube-api-access-jnqvs\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:56.905488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.905285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-serving-cert\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:56.905488 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:56.905310 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-oauth-serving-cert\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006156 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-service-ca\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqvs\" (UniqueName: \"kubernetes.io/projected/54a6416d-fadc-4fd9-80d4-51a89956dca3-kube-api-access-jnqvs\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-serving-cert\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-oauth-serving-cert\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-oauth-config\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006562 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-config\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.006942 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.006915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-config\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.007510 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.007484 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-service-ca\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.008794 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.008764 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-oauth-serving-cert\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.015672 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.015638 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-serving-cert\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.022764 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.022727 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-oauth-config\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.043052 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.035896 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqvs\" (UniqueName: \"kubernetes.io/projected/54a6416d-fadc-4fd9-80d4-51a89956dca3-kube-api-access-jnqvs\") pod \"console-8f7d455bf-fmsbb\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.056701 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.056658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 21:59:57.253581 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:57.253540 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f7d455bf-fmsbb"] Jul 07 21:59:58.775616 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:58.775571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:58.880414 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:58.880372 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:58.928590 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:58.928556 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww"] Jul 07 21:59:58.980835 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:58.980788 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:59.059241 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.059144 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww"] Jul 07 21:59:59.059241 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.059228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 21:59:59.061592 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.061567 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 21:59:59.061729 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.061575 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r6b72\"" Jul 07 21:59:59.227303 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.227256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/66b7f667-f0c4-4919-8ebc-95fd8e6b1015-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r5bww\" (UID: \"66b7f667-f0c4-4919-8ebc-95fd8e6b1015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 21:59:59.303406 ip-10-0-130-137 kubenswrapper[2565]: W0707 21:59:59.303373 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a6416d_fadc_4fd9_80d4_51a89956dca3.slice/crio-871949cd71c540d8a0c8da52ead0caab931912aa894708ed96cdf18282541be7 WatchSource:0}: Error finding container 871949cd71c540d8a0c8da52ead0caab931912aa894708ed96cdf18282541be7: Status 404 returned error can't find the container with id 871949cd71c540d8a0c8da52ead0caab931912aa894708ed96cdf18282541be7 Jul 07 21:59:59.328712 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.328629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/66b7f667-f0c4-4919-8ebc-95fd8e6b1015-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r5bww\" (UID: \"66b7f667-f0c4-4919-8ebc-95fd8e6b1015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 21:59:59.328855 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:59.328758 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 07 21:59:59.328855 ip-10-0-130-137 kubenswrapper[2565]: E0707 21:59:59.328830 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b7f667-f0c4-4919-8ebc-95fd8e6b1015-tls-certificates podName:66b7f667-f0c4-4919-8ebc-95fd8e6b1015 nodeName:}" failed. No retries permitted until 2026-07-07 21:59:59.828811438 +0000 UTC m=+55.820384644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/66b7f667-f0c4-4919-8ebc-95fd8e6b1015-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-r5bww" (UID: "66b7f667-f0c4-4919-8ebc-95fd8e6b1015") : secret "prometheus-operator-admission-webhook-tls" not found Jul 07 21:59:59.480237 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.480196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:59.559466 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.559431 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 21:59:59.831657 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.831608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/66b7f667-f0c4-4919-8ebc-95fd8e6b1015-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r5bww\" (UID: \"66b7f667-f0c4-4919-8ebc-95fd8e6b1015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 21:59:59.834442 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.834415 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/66b7f667-f0c4-4919-8ebc-95fd8e6b1015-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r5bww\" (UID: \"66b7f667-f0c4-4919-8ebc-95fd8e6b1015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 21:59:59.923473 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.923437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f7d455bf-fmsbb" event={"ID":"54a6416d-fadc-4fd9-80d4-51a89956dca3","Type":"ContainerStarted","Data":"871949cd71c540d8a0c8da52ead0caab931912aa894708ed96cdf18282541be7"} Jul 07 21:59:59.969823 ip-10-0-130-137 kubenswrapper[2565]: I0707 21:59:59.969774 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 22:00:01.747551 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.746744 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww"] Jul 07 22:00:01.757837 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:01.757793 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b7f667_f0c4_4919_8ebc_95fd8e6b1015.slice/crio-30eb4ff2493260c712030bf7d3a135eca23aecd4bd89620b138baf66b703386e WatchSource:0}: Error finding container 30eb4ff2493260c712030bf7d3a135eca23aecd4bd89620b138baf66b703386e: Status 404 returned error can't find the container with id 30eb4ff2493260c712030bf7d3a135eca23aecd4bd89620b138baf66b703386e Jul 07 22:00:01.931669 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.931214 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4s4fw" event={"ID":"71c673e5-2c40-47bb-92d5-0d73da843886","Type":"ContainerStarted","Data":"462253cc7aac4e18ee4aa580b412b71f571642cf2a5ff659571315f6d893dee9"} Jul 07 22:00:01.933424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.933393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9q6n" event={"ID":"8673fd68-bfb5-4761-950b-bdc1d1ac5020","Type":"ContainerStarted","Data":"43baaf73c98342cd516deebefbe2bfc8f55bd5b7aa33117fe365f99f5a51bf71"} Jul 07 22:00:01.935620 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.935584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46cgj" event={"ID":"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9","Type":"ContainerStarted","Data":"a9b73fe9066dd083ff8df1e41c1dac734280b84d354d90fd82c5318abcad3891"} Jul 07 22:00:01.935779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.935636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46cgj" event={"ID":"62047d1c-5895-4bda-b8a0-3a57fbc5eeb9","Type":"ContainerStarted","Data":"a276773a06032335a58e979131b2aeee4b2f9eb54046eed6eaf06571f544ec0a"} Jul 07 22:00:01.935779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.935710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-46cgj" Jul 07 22:00:01.937213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.937187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" event={"ID":"66b7f667-f0c4-4919-8ebc-95fd8e6b1015","Type":"ContainerStarted","Data":"30eb4ff2493260c712030bf7d3a135eca23aecd4bd89620b138baf66b703386e"} Jul 07 22:00:01.949832 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.949769 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4s4fw" podStartSLOduration=1.892214465 podStartE2EDuration="7.949751039s" totalImagesPullingTime="6.057536574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:01.949302592 +0000 UTC m=+57.940875831" watchObservedRunningTime="2026-07-07 22:00:01.949751039 +0000 UTC m=+57.941324268" Jul 07 22:00:01.970207 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:01.969217 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-46cgj" podStartSLOduration=1.8522415570000001 podStartE2EDuration="7.969196556s" totalImagesPullingTime="6.116954999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:01.96827393 +0000 UTC m=+57.959847159" watchObservedRunningTime="2026-07-07 22:00:01.969196556 +0000 UTC m=+57.960769787" Jul 07 22:00:03.944894 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.944860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f7d455bf-fmsbb" event={"ID":"54a6416d-fadc-4fd9-80d4-51a89956dca3","Type":"ContainerStarted","Data":"7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49"} Jul 07 22:00:03.946548 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.946514 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" event={"ID":"66b7f667-f0c4-4919-8ebc-95fd8e6b1015","Type":"ContainerStarted","Data":"1ca5faa89dda66615e3a454a2c1a7249237149c931bc03ca46692a9f0c126ec6"} Jul 07 22:00:03.946709 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.946582 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 22:00:03.948596 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.948565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9q6n" event={"ID":"8673fd68-bfb5-4761-950b-bdc1d1ac5020","Type":"ContainerStarted","Data":"856edec58a896799d59f214a2d7b9ededa98fd53f204d576dc136e619bf86784"} Jul 07 22:00:03.952234 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.952210 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" Jul 07 22:00:03.965151 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.965102 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8f7d455bf-fmsbb" podStartSLOduration=4.221110043 podStartE2EDuration="7.965085563s" totalImagesPullingTime="3.74397552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:03.96407948 +0000 UTC m=+59.955652710" watchObservedRunningTime="2026-07-07 22:00:03.965085563 +0000 UTC m=+59.956658791" Jul 07 22:00:03.983653 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:03.983601 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r5bww" podStartSLOduration=4.007423509 podStartE2EDuration="5.983583491s" totalImagesPullingTime="1.976159982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:03.982913933 +0000 UTC m=+59.974487163" watchObservedRunningTime="2026-07-07 22:00:03.983583491 +0000 UTC m=+59.975156720" Jul 07 22:00:04.959109 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.959075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:04.989731 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.989696 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8"] Jul 07 22:00:04.992979 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.992939 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:04.996872 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.996848 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 22:00:04.997016 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.996900 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-f2bm9\"" Jul 07 22:00:04.997993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.997787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 22:00:04.997993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.997817 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 22:00:04.998361 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.998344 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 22:00:04.998517 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:04.998385 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 22:00:05.003071 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.003046 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8"] Jul 07 22:00:05.040062 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.040011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:05.073264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.073224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.073264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.073265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqddv\" (UniqueName: \"kubernetes.io/projected/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-kube-api-access-zqddv\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.073619 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.073433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.073619 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.073481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.174620 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.174564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.174620 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.174621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqddv\" (UniqueName: \"kubernetes.io/projected/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-kube-api-access-zqddv\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.174886 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.174718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.174886 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.174745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.175439 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.175406 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.177705 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.177559 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.177705 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.177668 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.185173 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.185147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqddv\" (UniqueName: \"kubernetes.io/projected/e1ae39a6-fee6-45b5-93fb-9b6a82727a68-kube-api-access-zqddv\") pod \"prometheus-operator-567f7b8b7c-8psz8\" (UID: \"e1ae39a6-fee6-45b5-93fb-9b6a82727a68\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.263362 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.263324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:05.306451 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.306412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" Jul 07 22:00:05.334283 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.334248 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dbc45b887-j7j88"] Jul 07 22:00:05.338451 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.338411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.347847 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.347822 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 22:00:05.352367 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.352333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbc45b887-j7j88"] Jul 07 22:00:05.376425 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f772p\" (UniqueName: \"kubernetes.io/projected/ac5f117b-d131-47f0-8057-db5b823981d8-kube-api-access-f772p\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.376582 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-trusted-ca-bundle\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.376582 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-serving-cert\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.376691 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-oauth-config\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.376691 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376655 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-console-config\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.376792 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-oauth-serving-cert\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.376848 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.376798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-service-ca\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.458505 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.458465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8"] Jul 07 22:00:05.477319 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-serving-cert\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.477492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-oauth-config\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.477492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-console-config\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.477584 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-oauth-serving-cert\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.477584 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-service-ca\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.477676 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f772p\" (UniqueName: \"kubernetes.io/projected/ac5f117b-d131-47f0-8057-db5b823981d8-kube-api-access-f772p\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.477676 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.477628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-trusted-ca-bundle\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.478435 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.478310 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-console-config\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.478435 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.478310 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-oauth-serving-cert\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.478692 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.478669 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-service-ca\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.478787 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.478709 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-trusted-ca-bundle\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.480265 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.480243 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-oauth-config\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.480350 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.480325 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-serving-cert\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.489850 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.489824 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f772p\" (UniqueName: \"kubernetes.io/projected/ac5f117b-d131-47f0-8057-db5b823981d8-kube-api-access-f772p\") pod \"console-7dbc45b887-j7j88\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.650306 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.650212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:05.857760 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.857730 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm898" Jul 07 22:00:05.943411 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:05.943326 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:06.043408 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.043365 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:06.137437 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.137404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:06.595494 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:06.595460 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ae39a6_fee6_45b5_93fb_9b6a82727a68.slice/crio-f5efaab84011af43e51b4fdfb52e1bf88bcb08398b4baf93f42cb3b72b9ef2d4 WatchSource:0}: Error finding container f5efaab84011af43e51b4fdfb52e1bf88bcb08398b4baf93f42cb3b72b9ef2d4: Status 404 returned error can't find the container with id f5efaab84011af43e51b4fdfb52e1bf88bcb08398b4baf93f42cb3b72b9ef2d4 Jul 07 22:00:06.727534 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.727501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbc45b887-j7j88"] Jul 07 22:00:06.730999 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:06.730964 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5f117b_d131_47f0_8057_db5b823981d8.slice/crio-b11917f62cccf6d15c01592f77784e9c12d89e32f96d9ae521b4f782da49e546 WatchSource:0}: Error finding container b11917f62cccf6d15c01592f77784e9c12d89e32f96d9ae521b4f782da49e546: Status 404 returned error can't find the container with id b11917f62cccf6d15c01592f77784e9c12d89e32f96d9ae521b4f782da49e546 Jul 07 22:00:06.961198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.961097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc45b887-j7j88" event={"ID":"ac5f117b-d131-47f0-8057-db5b823981d8","Type":"ContainerStarted","Data":"08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120"} Jul 07 22:00:06.961198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.961163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc45b887-j7j88" event={"ID":"ac5f117b-d131-47f0-8057-db5b823981d8","Type":"ContainerStarted","Data":"b11917f62cccf6d15c01592f77784e9c12d89e32f96d9ae521b4f782da49e546"} Jul 07 22:00:06.962364 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.962328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" event={"ID":"e1ae39a6-fee6-45b5-93fb-9b6a82727a68","Type":"ContainerStarted","Data":"f5efaab84011af43e51b4fdfb52e1bf88bcb08398b4baf93f42cb3b72b9ef2d4"} Jul 07 22:00:06.979717 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:06.979673 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dbc45b887-j7j88" podStartSLOduration=1.9796546080000001 podStartE2EDuration="1.979654608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:06.978667183 +0000 UTC m=+62.970240399" watchObservedRunningTime="2026-07-07 22:00:06.979654608 +0000 UTC m=+62.971227813" Jul 07 22:00:07.057358 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.057268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 22:00:07.057725 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.057429 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 22:00:07.058971 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.058942 2565 patch_prober.go:28] interesting pod/console-8f7d455bf-fmsbb container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 07 22:00:07.059115 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.058993 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-8f7d455bf-fmsbb" podUID="54a6416d-fadc-4fd9-80d4-51a89956dca3" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 07 22:00:07.368620 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.368578 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:07.491327 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.491295 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:07.574704 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.574669 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:07.755052 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.755004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:07.876739 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.876708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:07.968924 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.968889 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9q6n" event={"ID":"8673fd68-bfb5-4761-950b-bdc1d1ac5020","Type":"ContainerStarted","Data":"48d3f63523a4158d32eaf8cc1ab19921d90df96fb8b2d7280424fe72242e529a"} Jul 07 22:00:07.995985 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:07.995922 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b9q6n" podStartSLOduration=2.665814981 podStartE2EDuration="13.995901809s" totalImagesPullingTime="11.330086828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:07.994834325 +0000 UTC m=+63.986407554" watchObservedRunningTime="2026-07-07 22:00:07.995901809 +0000 UTC m=+63.987475036" Jul 07 22:00:08.500447 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:08.500423 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:08.976006 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:08.975913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" event={"ID":"e1ae39a6-fee6-45b5-93fb-9b6a82727a68","Type":"ContainerStarted","Data":"c8fab6db763d532b16c86d21cc7cdb8e08d2611e33837cd1d19ebed640e5ed3d"} Jul 07 22:00:08.976006 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:08.975970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" event={"ID":"e1ae39a6-fee6-45b5-93fb-9b6a82727a68","Type":"ContainerStarted","Data":"be818daa81422899833d5b64ba34be2cf29aa61292e538253dfbf4aba24058f2"} Jul 07 22:00:08.994553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:08.994500 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-8psz8" podStartSLOduration=3.407547816 podStartE2EDuration="4.994482481s" totalImagesPullingTime="1.586934665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:08.993465366 +0000 UTC m=+64.985038596" watchObservedRunningTime="2026-07-07 22:00:08.994482481 +0000 UTC m=+64.986055708" Jul 07 22:00:10.114619 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.114573 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:10.195201 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.195164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:10.218161 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.218118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 22:00:10.218161 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.218168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 22:00:10.220745 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.220724 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 22:00:10.221301 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.221285 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 22:00:10.231527 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.231501 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17749b82-0e66-4e1d-a410-a6f414f06ff1-metrics-certs\") pod \"network-metrics-daemon-4zzjj\" (UID: \"17749b82-0e66-4e1d-a410-a6f414f06ff1\") " pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 22:00:10.231670 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.231581 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f9b71ef0-3555-4870-9392-4955495398c0-original-pull-secret\") pod \"global-pull-secret-syncer-cz4t2\" (UID: \"f9b71ef0-3555-4870-9392-4955495398c0\") " pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 22:00:10.389865 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.389776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cz4t2" Jul 07 22:00:10.406226 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.406198 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqsmx\"" Jul 07 22:00:10.414279 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.414249 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zzjj" Jul 07 22:00:10.420052 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.420013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 22:00:10.422277 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.422248 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 22:00:10.435845 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.435812 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 22:00:10.445713 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.445684 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq58x\" (UniqueName: \"kubernetes.io/projected/f64c2e17-a818-4c03-a33b-44817259494e-kube-api-access-gq58x\") pod \"network-check-target-qf59h\" (UID: \"f64c2e17-a818-4c03-a33b-44817259494e\") " pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 22:00:10.474920 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.474888 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:10.544334 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.544300 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cz4t2"] Jul 07 22:00:10.547676 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:10.547647 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b71ef0_3555_4870_9392_4955495398c0.slice/crio-12888b2122a476154480a838e8f49d28a0f5fd972dc087033684935d25e76498 WatchSource:0}: Error finding container 12888b2122a476154480a838e8f49d28a0f5fd972dc087033684935d25e76498: Status 404 returned error can't find the container with id 12888b2122a476154480a838e8f49d28a0f5fd972dc087033684935d25e76498 Jul 07 22:00:10.567197 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.567157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zzjj"] Jul 07 22:00:10.572340 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:10.572307 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17749b82_0e66_4e1d_a410_a6f414f06ff1.slice/crio-af989293a017ddf5783496b12602fb4f4151a11e5af355fcb84fdd37c7becfe9 WatchSource:0}: Error finding container af989293a017ddf5783496b12602fb4f4151a11e5af355fcb84fdd37c7becfe9: Status 404 returned error can't find the container with id af989293a017ddf5783496b12602fb4f4151a11e5af355fcb84fdd37c7becfe9 Jul 07 22:00:10.573230 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.573081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:10.712205 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.712131 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8gff\"" Jul 07 22:00:10.720759 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.720734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 22:00:10.851933 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.850213 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qf59h"] Jul 07 22:00:10.856417 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:10.856385 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64c2e17_a818_4c03_a33b_44817259494e.slice/crio-909a46762e3fc07b9ec1f0c0984e2c3eee017b9407face70198fa43594cb6d24 WatchSource:0}: Error finding container 909a46762e3fc07b9ec1f0c0984e2c3eee017b9407face70198fa43594cb6d24: Status 404 returned error can't find the container with id 909a46762e3fc07b9ec1f0c0984e2c3eee017b9407face70198fa43594cb6d24 Jul 07 22:00:10.983767 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.983662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qf59h" event={"ID":"f64c2e17-a818-4c03-a33b-44817259494e","Type":"ContainerStarted","Data":"909a46762e3fc07b9ec1f0c0984e2c3eee017b9407face70198fa43594cb6d24"} Jul 07 22:00:10.984850 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.984817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zzjj" event={"ID":"17749b82-0e66-4e1d-a410-a6f414f06ff1","Type":"ContainerStarted","Data":"af989293a017ddf5783496b12602fb4f4151a11e5af355fcb84fdd37c7becfe9"} Jul 07 22:00:10.985823 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:10.985799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cz4t2" event={"ID":"f9b71ef0-3555-4870-9392-4955495398c0","Type":"ContainerStarted","Data":"12888b2122a476154480a838e8f49d28a0f5fd972dc087033684935d25e76498"} Jul 07 22:00:11.110283 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.110238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:11.380275 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.380234 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs"] Jul 07 22:00:11.382565 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.382534 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.385778 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.385754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t29sw\"" Jul 07 22:00:11.386604 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.386580 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 22:00:11.386604 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.386594 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 22:00:11.392311 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.392284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:11.411116 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.410672 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs"] Jul 07 22:00:11.422319 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.422276 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cgs9n"] Jul 07 22:00:11.424764 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.424730 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427107 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427150 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kt52f\"" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427107 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78c08197-161f-41a0-8ed1-397c0eafef03-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c08197-161f-41a0-8ed1-397c0eafef03-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8x4\" (UniqueName: \"kubernetes.io/projected/78c08197-161f-41a0-8ed1-397c0eafef03-kube-api-access-qw8x4\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.427694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78c08197-161f-41a0-8ed1-397c0eafef03-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.428191 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.427943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 22:00:11.434636 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.434605 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk"] Jul 07 22:00:11.437096 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.437073 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.445380 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.445348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-n6qmv\"" Jul 07 22:00:11.445520 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.445378 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 22:00:11.445655 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.445638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 22:00:11.446850 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.446830 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 22:00:11.472713 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.472674 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk"] Jul 07 22:00:11.528658 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-tls\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.528658 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02da7917-ab60-4340-bbdf-cb39faf23ecc-metrics-client-ca\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-wtmp\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2abe1af1-919a-44da-8ab8-41598348bd64-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-textfile\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528858 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-sys\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78c08197-161f-41a0-8ed1-397c0eafef03-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.528939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c08197-161f-41a0-8ed1-397c0eafef03-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk79h\" (UniqueName: \"kubernetes.io/projected/02da7917-ab60-4340-bbdf-cb39faf23ecc-kube-api-access-tk79h\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.528997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8x4\" (UniqueName: \"kubernetes.io/projected/78c08197-161f-41a0-8ed1-397c0eafef03-kube-api-access-qw8x4\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-root\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.529281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2abe1af1-919a-44da-8ab8-41598348bd64-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.529658 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529291 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jprk\" (UniqueName: \"kubernetes.io/projected/2abe1af1-919a-44da-8ab8-41598348bd64-kube-api-access-8jprk\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.529658 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78c08197-161f-41a0-8ed1-397c0eafef03-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.529993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.529968 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78c08197-161f-41a0-8ed1-397c0eafef03-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.530442 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.530408 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:11.532209 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.532155 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78c08197-161f-41a0-8ed1-397c0eafef03-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.532425 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.532396 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c08197-161f-41a0-8ed1-397c0eafef03-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.538605 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.538577 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8x4\" (UniqueName: \"kubernetes.io/projected/78c08197-161f-41a0-8ed1-397c0eafef03-kube-api-access-qw8x4\") pod \"openshift-state-metrics-54cf96dfcf-q5wfs\" (UID: \"78c08197-161f-41a0-8ed1-397c0eafef03\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.629961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02da7917-ab60-4340-bbdf-cb39faf23ecc-metrics-client-ca\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-wtmp\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2abe1af1-919a-44da-8ab8-41598348bd64-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-textfile\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-sys\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk79h\" (UniqueName: \"kubernetes.io/projected/02da7917-ab60-4340-bbdf-cb39faf23ecc-kube-api-access-tk79h\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630182 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-wtmp\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-root\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2abe1af1-919a-44da-8ab8-41598348bd64-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jprk\" (UniqueName: \"kubernetes.io/projected/2abe1af1-919a-44da-8ab8-41598348bd64-kube-api-access-8jprk\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-tls\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.631405 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:11.630561 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:11.630622 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-tls podName:02da7917-ab60-4340-bbdf-cb39faf23ecc nodeName:}" failed. No retries permitted until 2026-07-07 22:00:12.130603129 +0000 UTC m=+68.122176350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-tls") pod "node-exporter-cgs9n" (UID: "02da7917-ab60-4340-bbdf-cb39faf23ecc") : secret "node-exporter-tls" not found Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630641 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02da7917-ab60-4340-bbdf-cb39faf23ecc-metrics-client-ca\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630823 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-sys\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.630921 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.631206 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-textfile\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.631287 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2abe1af1-919a-44da-8ab8-41598348bd64-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.631347 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02da7917-ab60-4340-bbdf-cb39faf23ecc-root\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.631783 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.632346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.631841 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2abe1af1-919a-44da-8ab8-41598348bd64-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.634937 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.634882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.635533 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.635507 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.636188 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.635934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:11.640757 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.640679 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk79h\" (UniqueName: \"kubernetes.io/projected/02da7917-ab60-4340-bbdf-cb39faf23ecc-kube-api-access-tk79h\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:11.641794 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.641774 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jprk\" (UniqueName: \"kubernetes.io/projected/2abe1af1-919a-44da-8ab8-41598348bd64-kube-api-access-8jprk\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.642784 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.642758 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2abe1af1-919a-44da-8ab8-41598348bd64-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-j28pk\" (UID: \"2abe1af1-919a-44da-8ab8-41598348bd64\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.694347 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.694307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" Jul 07 22:00:11.733923 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.733854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:11.748211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.748167 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" Jul 07 22:00:11.922613 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.922531 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:11.942167 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.941833 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-46cgj" Jul 07 22:00:11.956177 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.953887 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk"] Jul 07 22:00:11.960513 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:11.960338 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abe1af1_919a_44da_8ab8_41598348bd64.slice/crio-1d26d861945a3801b30853e751717e85525cb9ac98405cea6238e371ea0208f0 WatchSource:0}: Error finding container 1d26d861945a3801b30853e751717e85525cb9ac98405cea6238e371ea0208f0: Status 404 returned error can't find the container with id 1d26d861945a3801b30853e751717e85525cb9ac98405cea6238e371ea0208f0 Jul 07 22:00:11.978890 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:11.978862 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs"] Jul 07 22:00:11.984422 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:11.984394 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c08197_161f_41a0_8ed1_397c0eafef03.slice/crio-741a418bebbf168bffdb55036717d6448de6b72ed2c70016cbe7cef99933957a WatchSource:0}: Error finding container 741a418bebbf168bffdb55036717d6448de6b72ed2c70016cbe7cef99933957a: Status 404 returned error can't find the container with id 741a418bebbf168bffdb55036717d6448de6b72ed2c70016cbe7cef99933957a Jul 07 22:00:12.006320 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.006211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" event={"ID":"78c08197-161f-41a0-8ed1-397c0eafef03","Type":"ContainerStarted","Data":"741a418bebbf168bffdb55036717d6448de6b72ed2c70016cbe7cef99933957a"} Jul 07 22:00:12.012469 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.012361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" event={"ID":"2abe1af1-919a-44da-8ab8-41598348bd64","Type":"ContainerStarted","Data":"1d26d861945a3801b30853e751717e85525cb9ac98405cea6238e371ea0208f0"} Jul 07 22:00:12.016449 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.016357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zzjj" event={"ID":"17749b82-0e66-4e1d-a410-a6f414f06ff1","Type":"ContainerStarted","Data":"89892e80858424971146b117c58f680651bc69808f1c3d583fe485fa573e4b15"} Jul 07 22:00:12.025008 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.024976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:12.129186 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.129115 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:12.138197 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.137980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-tls\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:12.142698 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.142655 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02da7917-ab60-4340-bbdf-cb39faf23ecc-node-exporter-tls\") pod \"node-exporter-cgs9n\" (UID: \"02da7917-ab60-4340-bbdf-cb39faf23ecc\") " pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:12.336757 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.336717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cgs9n" Jul 07 22:00:12.338953 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.338913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:12.349042 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:12.348997 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02da7917_ab60_4340_bbdf_cb39faf23ecc.slice/crio-2f5fafa9cd7e6eebc08bee508a9b3195ccb851a371ffcaa4b935e1e371edc506 WatchSource:0}: Error finding container 2f5fafa9cd7e6eebc08bee508a9b3195ccb851a371ffcaa4b935e1e371edc506: Status 404 returned error can't find the container with id 2f5fafa9cd7e6eebc08bee508a9b3195ccb851a371ffcaa4b935e1e371edc506 Jul 07 22:00:12.459743 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.459658 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 22:00:12.462745 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.462663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:12.463480 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.463457 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.467168 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.467141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 22:00:12.467168 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.467161 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 22:00:12.467332 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.467181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 22:00:12.467332 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.467141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 22:00:12.467332 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.467141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 22:00:12.467996 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.467736 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 22:00:12.468140 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.468085 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v9kb6\"" Jul 07 22:00:12.468195 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.468168 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 22:00:12.468356 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.468342 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 22:00:12.468602 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.468581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 22:00:12.488233 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.488202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 22:00:12.542140 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542140 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-web-config\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2hx\" (UniqueName: \"kubernetes.io/projected/42ecc177-a45a-42f7-855f-17e2b6a13956-kube-api-access-bj2hx\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-config-volume\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542638 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42ecc177-a45a-42f7-855f-17e2b6a13956-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542638 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542638 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542638 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542548 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542638 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.542638 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.542596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42ecc177-a45a-42f7-855f-17e2b6a13956-config-out\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2hx\" (UniqueName: \"kubernetes.io/projected/42ecc177-a45a-42f7-855f-17e2b6a13956-kube-api-access-bj2hx\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643323 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643323 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643323 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-config-volume\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643323 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42ecc177-a45a-42f7-855f-17e2b6a13956-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42ecc177-a45a-42f7-855f-17e2b6a13956-config-out\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-web-config\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.643678 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.643477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.644993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.644964 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.645170 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:12.645147 2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 22:00:12.645281 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:12.645259 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-main-tls podName:42ecc177-a45a-42f7-855f-17e2b6a13956 nodeName:}" failed. No retries permitted until 2026-07-07 22:00:13.145205638 +0000 UTC m=+69.136778861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "42ecc177-a45a-42f7-855f-17e2b6a13956") : secret "alertmanager-main-tls" not found Jul 07 22:00:12.645418 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.645358 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.645583 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:12.645566 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-trusted-ca-bundle podName:42ecc177-a45a-42f7-855f-17e2b6a13956 nodeName:}" failed. No retries permitted until 2026-07-07 22:00:13.145546949 +0000 UTC m=+69.137120178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "42ecc177-a45a-42f7-855f-17e2b6a13956") : configmap references non-existent config key: ca-bundle.crt Jul 07 22:00:12.648829 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.648797 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.649405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.649355 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-web-config\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.649509 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.649462 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.649736 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.649713 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42ecc177-a45a-42f7-855f-17e2b6a13956-config-out\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.650515 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.650465 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42ecc177-a45a-42f7-855f-17e2b6a13956-tls-assets\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.650515 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.650472 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.650641 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.650590 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-config-volume\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.652002 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.651978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.652924 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.652903 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2hx\" (UniqueName: \"kubernetes.io/projected/42ecc177-a45a-42f7-855f-17e2b6a13956-kube-api-access-bj2hx\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:12.887059 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:12.886963 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:13.025370 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.025319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" event={"ID":"78c08197-161f-41a0-8ed1-397c0eafef03","Type":"ContainerStarted","Data":"71ecb78abf54a0545a6cb4df3d934601262bd4e242bcc62575ccb157a9a3b409"} Jul 07 22:00:13.025543 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.025399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" event={"ID":"78c08197-161f-41a0-8ed1-397c0eafef03","Type":"ContainerStarted","Data":"d690de4faa54131d0a5000aa379e66b0539a825552745593f9cc0eb0129ca967"} Jul 07 22:00:13.028282 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.028165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zzjj" event={"ID":"17749b82-0e66-4e1d-a410-a6f414f06ff1","Type":"ContainerStarted","Data":"72cd347e58828a55196cfb0d87765bbbb93fc0c585056099370c35bc2a9b9d90"} Jul 07 22:00:13.030331 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.030256 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgs9n" event={"ID":"02da7917-ab60-4340-bbdf-cb39faf23ecc","Type":"ContainerStarted","Data":"2f5fafa9cd7e6eebc08bee508a9b3195ccb851a371ffcaa4b935e1e371edc506"} Jul 07 22:00:13.054803 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.054745 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4zzjj" podStartSLOduration=67.826236769 podStartE2EDuration="1m9.054727142s" totalImagesPullingTime="1.228490373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:13.054410188 +0000 UTC m=+69.045983418" watchObservedRunningTime="2026-07-07 22:00:13.054727142 +0000 UTC m=+69.046300361" Jul 07 22:00:13.148692 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.148603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:13.148854 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.148701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:13.150208 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.150175 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42ecc177-a45a-42f7-855f-17e2b6a13956-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:13.155438 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.155374 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/42ecc177-a45a-42f7-855f-17e2b6a13956-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"42ecc177-a45a-42f7-855f-17e2b6a13956\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:13.219394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.219309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:13.334268 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.334181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:13.377837 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.377553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 22:00:13.465416 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.465319 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76f5c557b9-kgh6m"] Jul 07 22:00:13.488866 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.487866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.489501 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.489476 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76f5c557b9-kgh6m"] Jul 07 22:00:13.501873 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.501842 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-duttlnpkggnqo\"" Jul 07 22:00:13.502244 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.502223 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 22:00:13.503461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.502469 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tgs6j\"" Jul 07 22:00:13.503461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.502682 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 22:00:13.503461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.502847 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 22:00:13.503461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.503143 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 22:00:13.503461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.503328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 22:00:13.552137 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-tls\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552320 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-grpc-tls\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552320 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552337 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d2bc457-068c-407e-9864-92cbcd49e838-metrics-client-ca\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.552574 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.552496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn94x\" (UniqueName: \"kubernetes.io/projected/3d2bc457-068c-407e-9864-92cbcd49e838-kube-api-access-rn94x\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.653973 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.653914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-tls\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.653994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-grpc-tls\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.654043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.654066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.654108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.654148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.654165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d2bc457-068c-407e-9864-92cbcd49e838-metrics-client-ca\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.654515 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.654193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn94x\" (UniqueName: \"kubernetes.io/projected/3d2bc457-068c-407e-9864-92cbcd49e838-kube-api-access-rn94x\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.655675 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.655642 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d2bc457-068c-407e-9864-92cbcd49e838-metrics-client-ca\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.658299 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.658245 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-grpc-tls\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.658894 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.658845 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.660889 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.660861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.663653 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.663607 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.663790 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.663765 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-tls\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.665546 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.665519 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d2bc457-068c-407e-9864-92cbcd49e838-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.668003 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.667977 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn94x\" (UniqueName: \"kubernetes.io/projected/3d2bc457-068c-407e-9864-92cbcd49e838-kube-api-access-rn94x\") pod \"thanos-querier-76f5c557b9-kgh6m\" (UID: \"3d2bc457-068c-407e-9864-92cbcd49e838\") " pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:13.802107 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:13.802068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:14.234182 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:14.234063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:14.342749 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:14.342706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:14.686727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:14.686691 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:14.816989 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:14.816950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:14.919920 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:14.919881 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:15.449907 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.449870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:15.651347 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.651293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:15.651514 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.651386 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:15.657643 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.657608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:15.966572 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.966479 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78987745dc-qb8sq"] Jul 07 22:00:15.969553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.969522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:15.971891 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.971864 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 22:00:15.972048 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.971965 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 22:00:15.972269 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.972253 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 22:00:15.972346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.972250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eamrbmu82sgkr\"" Jul 07 22:00:15.972346 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.972330 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k7w82\"" Jul 07 22:00:15.972443 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.972352 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 22:00:15.982267 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:15.982235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78987745dc-qb8sq"] Jul 07 22:00:16.046815 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.046782 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:16.078070 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-secret-metrics-server-client-certs\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.078271 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078218 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-secret-metrics-server-tls\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.078383 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-client-ca-bundle\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.078430 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54q2\" (UniqueName: \"kubernetes.io/projected/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-kube-api-access-p54q2\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.078553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.078616 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-audit-log\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.078660 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.078610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-metrics-server-audit-profiles\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.155974 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.155945 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f7d455bf-fmsbb"] Jul 07 22:00:16.180018 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.179978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-client-ca-bundle\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54q2\" (UniqueName: \"kubernetes.io/projected/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-kube-api-access-p54q2\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-audit-log\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-metrics-server-audit-profiles\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-secret-metrics-server-client-certs\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180388 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-secret-metrics-server-tls\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.180871 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.180841 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-audit-log\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.182098 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.182071 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-metrics-server-audit-profiles\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.182640 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.182618 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.183485 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.183438 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-secret-metrics-server-tls\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.184781 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.184734 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-client-ca-bundle\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.185407 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.185381 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-secret-metrics-server-client-certs\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.194345 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.194313 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54q2\" (UniqueName: \"kubernetes.io/projected/d2bf5db6-ab88-4f7b-a3f9-2f012b420d50-kube-api-access-p54q2\") pod \"metrics-server-78987745dc-qb8sq\" (UID: \"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50\") " pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.251942 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.251895 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:16.281796 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.281535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:16.295700 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.295664 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n"] Jul 07 22:00:16.301214 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.301157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.304282 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.304256 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 22:00:16.304612 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.304589 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lhsgl\"" Jul 07 22:00:16.310588 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.310557 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n"] Jul 07 22:00:16.381709 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.381670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m62r\" (UniqueName: \"kubernetes.io/projected/08d529f5-c2d4-4a41-805b-20ba625634c7-kube-api-access-6m62r\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.381874 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.381730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08d529f5-c2d4-4a41-805b-20ba625634c7-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.432273 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.432075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:16.432676 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.432653 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54fd84ccf6-cv2vg"] Jul 07 22:00:16.435138 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.435114 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.448752 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.448723 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fd84ccf6-cv2vg"] Jul 07 22:00:16.483198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-serving-cert\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f54\" (UniqueName: \"kubernetes.io/projected/88413977-a2b3-4a70-b159-84da1558c8aa-kube-api-access-s8f54\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483442 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-console-config\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483442 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m62r\" (UniqueName: \"kubernetes.io/projected/08d529f5-c2d4-4a41-805b-20ba625634c7-kube-api-access-6m62r\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.483442 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-oauth-config\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483442 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-service-ca\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483649 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-trusted-ca-bundle\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483649 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08d529f5-c2d4-4a41-805b-20ba625634c7-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.483649 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.483539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-oauth-serving-cert\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.483772 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:16.483665 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 07 22:00:16.483772 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:16.483754 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08d529f5-c2d4-4a41-805b-20ba625634c7-monitoring-plugin-cert podName:08d529f5-c2d4-4a41-805b-20ba625634c7 nodeName:}" failed. No retries permitted until 2026-07-07 22:00:16.983728871 +0000 UTC m=+72.975302077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/08d529f5-c2d4-4a41-805b-20ba625634c7-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-rlb9n" (UID: "08d529f5-c2d4-4a41-805b-20ba625634c7") : secret "monitoring-plugin-cert" not found Jul 07 22:00:16.493317 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.493289 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m62r\" (UniqueName: \"kubernetes.io/projected/08d529f5-c2d4-4a41-805b-20ba625634c7-kube-api-access-6m62r\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.584603 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-oauth-config\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.584603 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-service-ca\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.584603 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-trusted-ca-bundle\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.584959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-oauth-serving-cert\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.584959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-serving-cert\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.584959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f54\" (UniqueName: \"kubernetes.io/projected/88413977-a2b3-4a70-b159-84da1558c8aa-kube-api-access-s8f54\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.584959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.584762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-console-config\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.585504 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.585389 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-service-ca\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.586155 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.586098 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-console-config\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.586286 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.586169 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-oauth-serving-cert\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.586286 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.586242 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-trusted-ca-bundle\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.587547 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.587522 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-oauth-config\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.587717 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.587693 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-serving-cert\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.596914 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.596886 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f54\" (UniqueName: \"kubernetes.io/projected/88413977-a2b3-4a70-b159-84da1558c8aa-kube-api-access-s8f54\") pod \"console-54fd84ccf6-cv2vg\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.658068 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.658007 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m"] Jul 07 22:00:16.661521 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.661492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.662887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.662856 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:16.664205 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.664180 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 22:00:16.664518 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.664494 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wklwk\"" Jul 07 22:00:16.664716 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.664699 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 22:00:16.664913 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.664899 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 22:00:16.665087 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.665072 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 22:00:16.665281 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.665264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 22:00:16.672131 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.671361 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 22:00:16.679534 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.679501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m"] Jul 07 22:00:16.745928 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.745669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:16.786860 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.786815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzk8\" (UniqueName: \"kubernetes.io/projected/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-kube-api-access-xrzk8\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787053 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.786881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-telemeter-client-tls\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787053 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.786964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.787089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-metrics-client-ca\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.787120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-secret-telemeter-client\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787277 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.787174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-federate-client-tls\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787277 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.787231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-serving-certs-ca-bundle\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.787277 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.787261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888257 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-metrics-client-ca\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888257 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-secret-telemeter-client\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-federate-client-tls\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-serving-certs-ca-bundle\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzk8\" (UniqueName: \"kubernetes.io/projected/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-kube-api-access-xrzk8\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888700 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-telemeter-client-tls\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.888700 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.888624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.889211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.889184 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-serving-certs-ca-bundle\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.889359 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.889240 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-metrics-client-ca\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.889487 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.889466 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.891306 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.891280 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-secret-telemeter-client\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.891436 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.891349 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-telemeter-client-tls\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.891495 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.891461 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.891715 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.891697 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-federate-client-tls\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.897770 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.897740 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzk8\" (UniqueName: \"kubernetes.io/projected/8b3bc5bf-dc00-44d6-aab2-8af876b3a17d-kube-api-access-xrzk8\") pod \"telemeter-client-6dcf7b559b-ppb5m\" (UID: \"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d\") " pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.974881 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.974835 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" Jul 07 22:00:16.990001 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.989960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08d529f5-c2d4-4a41-805b-20ba625634c7-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:16.992683 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:16.992646 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08d529f5-c2d4-4a41-805b-20ba625634c7-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-rlb9n\" (UID: \"08d529f5-c2d4-4a41-805b-20ba625634c7\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:17.189582 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.189543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:17.215110 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.214893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:17.293405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.293304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:17.461556 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.461515 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76f5c557b9-kgh6m"] Jul 07 22:00:17.541978 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.540901 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 22:00:17.560539 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:17.560475 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ecc177_a45a_42f7_855f_17e2b6a13956.slice/crio-06980c37b2e4fc3dd7b01bf2455f4228428980af83d102c3766804015b70e757 WatchSource:0}: Error finding container 06980c37b2e4fc3dd7b01bf2455f4228428980af83d102c3766804015b70e757: Status 404 returned error can't find the container with id 06980c37b2e4fc3dd7b01bf2455f4228428980af83d102c3766804015b70e757 Jul 07 22:00:17.567353 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.565931 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78987745dc-qb8sq"] Jul 07 22:00:17.584112 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:17.584064 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bf5db6_ab88_4f7b_a3f9_2f012b420d50.slice/crio-e07149326671c76f2843934de37113c135337acb344a5589ce2364c453a3980a WatchSource:0}: Error finding container e07149326671c76f2843934de37113c135337acb344a5589ce2364c453a3980a: Status 404 returned error can't find the container with id e07149326671c76f2843934de37113c135337acb344a5589ce2364c453a3980a Jul 07 22:00:17.610357 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.610321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n"] Jul 07 22:00:17.630791 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:17.630759 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d529f5_c2d4_4a41_805b_20ba625634c7.slice/crio-0b3a03573396a0ed44741dade0edaa13a011a3169a8190b2e3f7d7039e8a5214 WatchSource:0}: Error finding container 0b3a03573396a0ed44741dade0edaa13a011a3169a8190b2e3f7d7039e8a5214: Status 404 returned error can't find the container with id 0b3a03573396a0ed44741dade0edaa13a011a3169a8190b2e3f7d7039e8a5214 Jul 07 22:00:17.700175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.700145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fd84ccf6-cv2vg"] Jul 07 22:00:17.701632 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.701537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m"] Jul 07 22:00:17.716540 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:17.716507 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88413977_a2b3_4a70_b159_84da1558c8aa.slice/crio-247dba65a299c7de49cfd83f5610898c8986badd835c51698c7da44b9272ae3d WatchSource:0}: Error finding container 247dba65a299c7de49cfd83f5610898c8986badd835c51698c7da44b9272ae3d: Status 404 returned error can't find the container with id 247dba65a299c7de49cfd83f5610898c8986badd835c51698c7da44b9272ae3d Jul 07 22:00:17.716716 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:17.716695 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3bc5bf_dc00_44d6_aab2_8af876b3a17d.slice/crio-bc950bf79ca7ac81da045c966aa9a775a39af7443052940bc582f4cda95360f1 WatchSource:0}: Error finding container bc950bf79ca7ac81da045c966aa9a775a39af7443052940bc582f4cda95360f1: Status 404 returned error can't find the container with id bc950bf79ca7ac81da045c966aa9a775a39af7443052940bc582f4cda95360f1 Jul 07 22:00:17.758240 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.758195 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:17.863593 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.863546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:17.903102 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.902995 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 22:00:17.906596 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.906566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:17.910999 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.910487 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 22:00:17.910999 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.910517 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 22:00:17.910999 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.910814 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 22:00:17.910999 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.910814 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 22:00:17.911319 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.911118 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 22:00:17.911370 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.911324 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 22:00:17.911419 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.911377 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 22:00:17.911787 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.911618 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 22:00:17.912844 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.912819 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 22:00:17.912937 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.912857 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f83p6vk9egb20\"" Jul 07 22:00:17.913089 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.913065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 22:00:17.913297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.913276 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 22:00:17.913385 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.913313 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7zc5\"" Jul 07 22:00:17.913516 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.913489 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 22:00:17.917599 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.917573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 22:00:17.923190 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.923158 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 22:00:17.928820 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.928772 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 22:00:17.960751 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:17.960715 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:18.011227 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-web-config\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011448 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52d2d8a8-c7fe-4ffb-89f0-72732478793a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-config\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52d2d8a8-c7fe-4ffb-89f0-72732478793a-config-out\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.011723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62rj\" (UniqueName: \"kubernetes.io/projected/52d2d8a8-c7fe-4ffb-89f0-72732478793a-kube-api-access-b62rj\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.011970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.012088 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.012118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.012210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.012238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.012583 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.012304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.053322 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.053280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cz4t2" event={"ID":"f9b71ef0-3555-4870-9392-4955495398c0","Type":"ContainerStarted","Data":"8077cd0c1c30a415c549c5ccc1e4c1608d1fe816d4edf0a3d900d7565442af41"} Jul 07 22:00:18.055085 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.055050 2565 generic.go:356] "Generic (PLEG): container finished" podID="02da7917-ab60-4340-bbdf-cb39faf23ecc" containerID="3b34469fa2ab8cbcc5d40aa7fb1b23f21394fdde150bd8a95f70af6026b5a32c" exitCode=0 Jul 07 22:00:18.055235 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.055130 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgs9n" event={"ID":"02da7917-ab60-4340-bbdf-cb39faf23ecc","Type":"ContainerDied","Data":"3b34469fa2ab8cbcc5d40aa7fb1b23f21394fdde150bd8a95f70af6026b5a32c"} Jul 07 22:00:18.056487 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.056456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" event={"ID":"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50","Type":"ContainerStarted","Data":"e07149326671c76f2843934de37113c135337acb344a5589ce2364c453a3980a"} Jul 07 22:00:18.057737 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.057714 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" event={"ID":"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d","Type":"ContainerStarted","Data":"bc950bf79ca7ac81da045c966aa9a775a39af7443052940bc582f4cda95360f1"} Jul 07 22:00:18.059580 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.059557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qf59h" event={"ID":"f64c2e17-a818-4c03-a33b-44817259494e","Type":"ContainerStarted","Data":"57e24c79cb8002026be11c48bc2e6f0c9ca323b5545fe73a434e4e5ef84b1dd9"} Jul 07 22:00:18.059721 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.059700 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 22:00:18.061843 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.061820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" event={"ID":"78c08197-161f-41a0-8ed1-397c0eafef03","Type":"ContainerStarted","Data":"311c9713650566cc92ca23e3f0b285a1033d82721f08f08bf2fcfbc23a35270f"} Jul 07 22:00:18.064260 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.064234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" event={"ID":"2abe1af1-919a-44da-8ab8-41598348bd64","Type":"ContainerStarted","Data":"97c1294216a2470b68923f8d3782882e29762fbff37e4692233779b9b78843d0"} Jul 07 22:00:18.064373 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.064297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" event={"ID":"2abe1af1-919a-44da-8ab8-41598348bd64","Type":"ContainerStarted","Data":"a6bb2fe5633d76b4cd49b44edda6a606830c3abb18a96f6f9fb08e799565ddea"} Jul 07 22:00:18.064373 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.064326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" event={"ID":"2abe1af1-919a-44da-8ab8-41598348bd64","Type":"ContainerStarted","Data":"8a1f6de6f8ff133166f608c3cbb405640c59459cf379e196a517c494321fa1bd"} Jul 07 22:00:18.065460 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.065440 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" event={"ID":"08d529f5-c2d4-4a41-805b-20ba625634c7","Type":"ContainerStarted","Data":"0b3a03573396a0ed44741dade0edaa13a011a3169a8190b2e3f7d7039e8a5214"} Jul 07 22:00:18.067088 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.067062 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fd84ccf6-cv2vg" event={"ID":"88413977-a2b3-4a70-b159-84da1558c8aa","Type":"ContainerStarted","Data":"2c92a7168afff549984311bbdd5b41151a2749fc9b9401e035bb8170d17655e1"} Jul 07 22:00:18.067233 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.067109 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fd84ccf6-cv2vg" event={"ID":"88413977-a2b3-4a70-b159-84da1558c8aa","Type":"ContainerStarted","Data":"247dba65a299c7de49cfd83f5610898c8986badd835c51698c7da44b9272ae3d"} Jul 07 22:00:18.068334 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.068311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"06980c37b2e4fc3dd7b01bf2455f4228428980af83d102c3766804015b70e757"} Jul 07 22:00:18.069511 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.069489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"399197447d38ebf57bffb67e1b8c002d037c83500412fb04290b5fd0241e6398"} Jul 07 22:00:18.072229 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.072158 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cz4t2" podStartSLOduration=67.431266208 podStartE2EDuration="1m14.072123831s" totalImagesPullingTime="6.640857623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:18.072099793 +0000 UTC m=+74.063673019" watchObservedRunningTime="2026-07-07 22:00:18.072123831 +0000 UTC m=+74.063697056" Jul 07 22:00:18.093715 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.093664 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-q5wfs" podStartSLOduration=2.117570286 podStartE2EDuration="7.09364905s" totalImagesPullingTime="4.976078764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:18.093287266 +0000 UTC m=+74.084860496" watchObservedRunningTime="2026-07-07 22:00:18.09364905 +0000 UTC m=+74.085222310" Jul 07 22:00:18.113912 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.113875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114099 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.113933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52d2d8a8-c7fe-4ffb-89f0-72732478793a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114099 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.113980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114099 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114099 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-config\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114099 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114362 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52d2d8a8-c7fe-4ffb-89f0-72732478793a-config-out\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114362 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114453 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62rj\" (UniqueName: \"kubernetes.io/projected/52d2d8a8-c7fe-4ffb-89f0-72732478793a-kube-api-access-b62rj\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114506 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114506 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114595 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114595 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114697 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114697 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114697 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114818 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114818 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-web-config\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114907 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.114907 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.114878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.120444 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.119052 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.120444 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.119856 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.121135 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.120691 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.121458 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.121429 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.121553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.121501 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.126213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.123969 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.126213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.124449 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-web-config\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.126213 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.124958 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.129765 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.129719 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.130711 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.130126 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.136060 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.134749 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.138993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.137899 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.138993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.138274 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.138993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.138519 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62rj\" (UniqueName: \"kubernetes.io/projected/52d2d8a8-c7fe-4ffb-89f0-72732478793a-kube-api-access-b62rj\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.139343 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.139300 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/52d2d8a8-c7fe-4ffb-89f0-72732478793a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.139729 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.139664 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52d2d8a8-c7fe-4ffb-89f0-72732478793a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.140080 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.140001 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.140264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.140190 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.140886 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.140867 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/52d2d8a8-c7fe-4ffb-89f0-72732478793a-config\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.141522 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.141496 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52d2d8a8-c7fe-4ffb-89f0-72732478793a-config-out\") pod \"prometheus-k8s-0\" (UID: \"52d2d8a8-c7fe-4ffb-89f0-72732478793a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.152841 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.152775 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j28pk" podStartSLOduration=1.954016191 podStartE2EDuration="7.152749978s" totalImagesPullingTime="5.198733787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:18.151772039 +0000 UTC m=+74.143345273" watchObservedRunningTime="2026-07-07 22:00:18.152749978 +0000 UTC m=+74.144323207" Jul 07 22:00:18.200148 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.199841 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qf59h" podStartSLOduration=67.89085589 podStartE2EDuration="1m14.199816894s" totalImagesPullingTime="6.308961004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:18.172345573 +0000 UTC m=+74.163918791" watchObservedRunningTime="2026-07-07 22:00:18.199816894 +0000 UTC m=+74.191390122" Jul 07 22:00:18.201500 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.201215 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54fd84ccf6-cv2vg" podStartSLOduration=2.20120146 podStartE2EDuration="2.20120146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:18.200716236 +0000 UTC m=+74.192289464" watchObservedRunningTime="2026-07-07 22:00:18.20120146 +0000 UTC m=+74.192774666" Jul 07 22:00:18.222072 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.222011 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:18.417832 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.417739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 22:00:18.422365 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:18.422321 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d2d8a8_c7fe_4ffb_89f0_72732478793a.slice/crio-ecce77921fa4e05fd7f96f88b3a1dba38b8f66becbd82267c7485624634e6b97 WatchSource:0}: Error finding container ecce77921fa4e05fd7f96f88b3a1dba38b8f66becbd82267c7485624634e6b97: Status 404 returned error can't find the container with id ecce77921fa4e05fd7f96f88b3a1dba38b8f66becbd82267c7485624634e6b97 Jul 07 22:00:18.949094 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:18.948968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:19.047081 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:19.046977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:19.079129 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:19.077504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"ecce77921fa4e05fd7f96f88b3a1dba38b8f66becbd82267c7485624634e6b97"} Jul 07 22:00:19.086055 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:19.083009 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgs9n" event={"ID":"02da7917-ab60-4340-bbdf-cb39faf23ecc","Type":"ContainerStarted","Data":"d9a23a002c4d0f3e2725290458f264dbdf9fa7cc7f6d8418badf4bda2d0967e5"} Jul 07 22:00:19.086055 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:19.083094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cgs9n" event={"ID":"02da7917-ab60-4340-bbdf-cb39faf23ecc","Type":"ContainerStarted","Data":"c7f0a4af4c723b6b159659d6c720a4292bb91c889a96f95111bc7acf3b0c7fdb"} Jul 07 22:00:19.103384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:19.103330 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cgs9n" podStartSLOduration=3.129331667 podStartE2EDuration="8.103312043s" totalImagesPullingTime="4.807567101s" totalInitContainerRuntime="166.413275ms" isStatefulPod=true podCreationTimestamp="2026-07-07 22:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:19.103001688 +0000 UTC m=+75.094574916" watchObservedRunningTime="2026-07-07 22:00:19.103312043 +0000 UTC m=+75.094885271" Jul 07 22:00:20.088595 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:20.088552 2565 generic.go:356] "Generic (PLEG): container finished" podID="52d2d8a8-c7fe-4ffb-89f0-72732478793a" containerID="259947e3d5dd869fd56884696369655a0a124a03e37fa52eb58e3b9e6db66e18" exitCode=0 Jul 07 22:00:20.089046 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:20.088654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerDied","Data":"259947e3d5dd869fd56884696369655a0a124a03e37fa52eb58e3b9e6db66e18"} Jul 07 22:00:20.090995 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:20.090968 2565 generic.go:356] "Generic (PLEG): container finished" podID="42ecc177-a45a-42f7-855f-17e2b6a13956" containerID="749f9bcd1cc717f61307cf2d8fceba655e5e63bfb8dd5d89cbe08ae025bc8723" exitCode=0 Jul 07 22:00:20.091111 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:20.091064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerDied","Data":"749f9bcd1cc717f61307cf2d8fceba655e5e63bfb8dd5d89cbe08ae025bc8723"} Jul 07 22:00:22.097603 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.097557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:22.103757 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.103712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" event={"ID":"08d529f5-c2d4-4a41-805b-20ba625634c7","Type":"ContainerStarted","Data":"6b87d7ab7334fde50157f3a9ef5bc8bacc73b5bf68ad2d410a364d4644b399cf"} Jul 07 22:00:22.104935 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.104861 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:22.110064 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.109883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"00111612aa87239629a3415dcbe23f7642f2a7b2286766029f1abafe8076f9fd"} Jul 07 22:00:22.110064 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.109960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"32a9199366c6d40101d831747e9c307cc95b4cd100f682000965b6b0bde1d592"} Jul 07 22:00:22.110064 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.109990 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"2fdea7732a6171eb662df89799acb9b0229b609a6e8cfe31b5010c1463692d70"} Jul 07 22:00:22.111699 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.111671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" Jul 07 22:00:22.112440 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.112407 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" event={"ID":"d2bf5db6-ab88-4f7b-a3f9-2f012b420d50","Type":"ContainerStarted","Data":"cbf112b970c377909dce016fcd94a7c192c01cc804e63fc0270b47d559410c90"} Jul 07 22:00:22.117191 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.116383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" event={"ID":"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d","Type":"ContainerStarted","Data":"fbcf745b52402550badf65957c1820fda0eb80045cd898702a2981d9f47bb453"} Jul 07 22:00:22.117191 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.116443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" event={"ID":"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d","Type":"ContainerStarted","Data":"c784e1622168491e17d55453e4afa6659d0734bcdff2dbf7ac43d9f5acdfd6c8"} Jul 07 22:00:22.117191 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.116471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" event={"ID":"8b3bc5bf-dc00-44d6-aab2-8af876b3a17d","Type":"ContainerStarted","Data":"fc66f8ee828da5a870f5d1e24b1c7c7e2edddfba7b21d0d8893d9695c697ecad"} Jul 07 22:00:22.124063 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.122932 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-rlb9n" podStartSLOduration=2.430309974 podStartE2EDuration="6.122908159s" totalImagesPullingTime="3.692598185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:22.121741157 +0000 UTC m=+78.113314386" watchObservedRunningTime="2026-07-07 22:00:22.122908159 +0000 UTC m=+78.114481389" Jul 07 22:00:22.141681 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.141619 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" podStartSLOduration=3.40184635 podStartE2EDuration="7.141599298s" totalImagesPullingTime="3.739752948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:22.139922 +0000 UTC m=+78.131495224" watchObservedRunningTime="2026-07-07 22:00:22.141599298 +0000 UTC m=+78.133172526" Jul 07 22:00:22.181563 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.181483 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dcf7b559b-ppb5m" podStartSLOduration=2.5723251449999998 podStartE2EDuration="6.181460615s" totalImagesPullingTime="3.60913547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:22.179344559 +0000 UTC m=+78.170917822" watchObservedRunningTime="2026-07-07 22:00:22.181460615 +0000 UTC m=+78.173033842" Jul 07 22:00:22.186304 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:22.186268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:23.108522 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.108489 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fd84ccf6-cv2vg"] Jul 07 22:00:23.130379 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.130172 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"64977f5e1a029f621adebb59d73e835b82da067d394b0b1f2ac0853299c7bc01"} Jul 07 22:00:23.130379 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.130260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"bae10975583226e2bcb80f80c4776308616f3ca579a55f5c973f112b91a8bf85"} Jul 07 22:00:23.135604 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.135571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"2540a4523cfd168bc289a0e2ba1a2c3c168572e9b622857b77d3fc821e2b62eb"} Jul 07 22:00:23.135919 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.135901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"8e43862638546284f92b0899644b446e7eac3972ee2f8e2518b388ebd8f08fbb"} Jul 07 22:00:23.153648 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.153623 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dd897c74c-rss2g"] Jul 07 22:00:23.160286 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.160261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.172974 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.172940 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd897c74c-rss2g"] Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-oauth-config\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-oauth-serving-cert\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-trusted-ca-bundle\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-config\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c542p\" (UniqueName: \"kubernetes.io/projected/2549ac44-daf5-47b3-942d-8efcce6d82aa-kube-api-access-c542p\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-serving-cert\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.290459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.289591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-service-ca\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.324488 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.324452 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:23.390537 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-serving-cert\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.390537 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-service-ca\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.390783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-oauth-config\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.390783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-oauth-serving-cert\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.390783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-trusted-ca-bundle\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.390783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-config\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.390783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.390720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c542p\" (UniqueName: \"kubernetes.io/projected/2549ac44-daf5-47b3-942d-8efcce6d82aa-kube-api-access-c542p\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.391341 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.391317 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-service-ca\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.391502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.391429 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-oauth-serving-cert\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.391502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.391486 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-config\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.391649 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.391624 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-trusted-ca-bundle\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.393938 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.393909 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-oauth-config\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.394043 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.393991 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-serving-cert\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.399930 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.399901 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c542p\" (UniqueName: \"kubernetes.io/projected/2549ac44-daf5-47b3-942d-8efcce6d82aa-kube-api-access-c542p\") pod \"console-dd897c74c-rss2g\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.423591 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.423486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:23.478783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.478747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:23.838375 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.838338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:23.937470 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:23.937422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:24.145184 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.145076 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"f8b6ef5e4e8657e34b85e7e65446c20394e3aff73ac1ae9d03764328010308e0"} Jul 07 22:00:24.145577 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.145213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"5d228917fb1db1d60129b92317af2b64812dbe12948e8f5016d89b0ab803682f"} Jul 07 22:00:24.145577 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.145253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"52d7aa2ea049b70b83c977c0ec15510419119cb3c434e426195c0024d8d2b43c"} Jul 07 22:00:24.149258 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.149222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" event={"ID":"3d2bc457-068c-407e-9864-92cbcd49e838","Type":"ContainerStarted","Data":"3b6209b2aca45e715bf12d63eb52dbd3b84669c2cd1fbc8e0d743330cb8b961b"} Jul 07 22:00:24.176141 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.176081 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" podStartSLOduration=6.033395088 podStartE2EDuration="11.176060447s" totalImagesPullingTime="5.142665359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:24.174604521 +0000 UTC m=+80.166177750" watchObservedRunningTime="2026-07-07 22:00:24.176060447 +0000 UTC m=+80.167633676" Jul 07 22:00:24.461311 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.461276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:24.586476 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:24.586445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd897c74c-rss2g"] Jul 07 22:00:24.590376 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:00:24.590350 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2549ac44_daf5_47b3_942d_8efcce6d82aa.slice/crio-a3e50d6548171bb1d1b462a24470d95be49084bccf1f565ee29d6a9244fd42ba WatchSource:0}: Error finding container a3e50d6548171bb1d1b462a24470d95be49084bccf1f565ee29d6a9244fd42ba: Status 404 returned error can't find the container with id a3e50d6548171bb1d1b462a24470d95be49084bccf1f565ee29d6a9244fd42ba Jul 07 22:00:25.157772 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.157729 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"42ecc177-a45a-42f7-855f-17e2b6a13956","Type":"ContainerStarted","Data":"ed281b985ec23f4fc5129c2ff864f91cd95fe5f2ba4d86da651a2c78a428119c"} Jul 07 22:00:25.159407 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.159380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd897c74c-rss2g" event={"ID":"2549ac44-daf5-47b3-942d-8efcce6d82aa","Type":"ContainerStarted","Data":"8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39"} Jul 07 22:00:25.159536 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.159444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd897c74c-rss2g" event={"ID":"2549ac44-daf5-47b3-942d-8efcce6d82aa","Type":"ContainerStarted","Data":"a3e50d6548171bb1d1b462a24470d95be49084bccf1f565ee29d6a9244fd42ba"} Jul 07 22:00:25.162899 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.162870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"8df024a5d51d3710496113fadf875497de31b8bf06c1593642099a8be6d17a86"} Jul 07 22:00:25.163002 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.162946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"0d6aa03364ef95f20095b34adbdc71165d2987385ac79d01b0ff49e1ad8cabc8"} Jul 07 22:00:25.163002 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.162986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"a190d225c0e9dc04481b773dc00b1cb62fc0538bbb3a85f71ccb7ffe82768675"} Jul 07 22:00:25.163112 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.163017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"72c5a4501855f9435fce333dc34699356fad1da39cf372fe4d8774c74a94d201"} Jul 07 22:00:25.163112 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.163075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"dd86be4b1df9200aea1f45a2ddce7dd7f92c077e1d5a5fdb774c6669cfcd9b1a"} Jul 07 22:00:25.163201 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.163184 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:25.163257 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.163242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"52d2d8a8-c7fe-4ffb-89f0-72732478793a","Type":"ContainerStarted","Data":"7bb64048e32fa4a1674976c36f914ca3193a6f20d4e21296473b8001f577440b"} Jul 07 22:00:25.187692 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.187568 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.85763793 podStartE2EDuration="13.187544781s" totalImagesPullingTime="4.301406992s" totalInitContainerRuntime="28.499859ms" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:25.185867184 +0000 UTC m=+81.177440470" watchObservedRunningTime="2026-07-07 22:00:25.187544781 +0000 UTC m=+81.179118011" Jul 07 22:00:25.214264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.213953 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dd897c74c-rss2g" podStartSLOduration=2.213928304 podStartE2EDuration="2.213928304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:25.212856333 +0000 UTC m=+81.204429562" watchObservedRunningTime="2026-07-07 22:00:25.213928304 +0000 UTC m=+81.205501570" Jul 07 22:00:25.244769 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:25.244701 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.087069314 podStartE2EDuration="8.244678939s" totalImagesPullingTime="5.128250446s" totalInitContainerRuntime="29.359179ms" isStatefulPod=false podCreationTimestamp="2026-07-07 22:00:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:00:25.242763342 +0000 UTC m=+81.234336571" watchObservedRunningTime="2026-07-07 22:00:25.244678939 +0000 UTC m=+81.236252168" Jul 07 22:00:26.174007 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:26.173972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76f5c557b9-kgh6m" Jul 07 22:00:26.746799 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:26.746756 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:27.228766 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:27.228725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:27.335358 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:27.335325 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:28.223097 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:28.223057 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:00:32.869745 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:32.869708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:33.002627 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:33.002591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:33.479210 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:33.479178 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:33.479384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:33.479243 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:33.483864 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:33.483841 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:34.208093 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:34.208061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:00:34.257664 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:34.257632 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbc45b887-j7j88"] Jul 07 22:00:34.388690 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:34.388654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:34.489841 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:34.489736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:36.281962 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:36.281928 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:36.282371 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:36.282041 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:37.108385 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:37.108350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:37.202516 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:37.202463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:41.185641 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.185570 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8f7d455bf-fmsbb" podUID="54a6416d-fadc-4fd9-80d4-51a89956dca3" containerName="console" containerID="cri-o://7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49" gracePeriod=15 Jul 07 22:00:41.432065 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.432040 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 22:00:41.456680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.456587 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-serving-cert\") pod \"54a6416d-fadc-4fd9-80d4-51a89956dca3\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " Jul 07 22:00:41.456680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.456650 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-oauth-serving-cert\") pod \"54a6416d-fadc-4fd9-80d4-51a89956dca3\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " Jul 07 22:00:41.456898 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.456696 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-config\") pod \"54a6416d-fadc-4fd9-80d4-51a89956dca3\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " Jul 07 22:00:41.456898 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.456761 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnqvs\" (UniqueName: \"kubernetes.io/projected/54a6416d-fadc-4fd9-80d4-51a89956dca3-kube-api-access-jnqvs\") pod \"54a6416d-fadc-4fd9-80d4-51a89956dca3\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " Jul 07 22:00:41.456898 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.456811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-oauth-config\") pod \"54a6416d-fadc-4fd9-80d4-51a89956dca3\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " Jul 07 22:00:41.456898 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.456842 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-service-ca\") pod \"54a6416d-fadc-4fd9-80d4-51a89956dca3\" (UID: \"54a6416d-fadc-4fd9-80d4-51a89956dca3\") " Jul 07 22:00:41.457190 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.457158 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "54a6416d-fadc-4fd9-80d4-51a89956dca3" (UID: "54a6416d-fadc-4fd9-80d4-51a89956dca3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:41.457430 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.457399 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-service-ca" (OuterVolumeSpecName: "service-ca") pod "54a6416d-fadc-4fd9-80d4-51a89956dca3" (UID: "54a6416d-fadc-4fd9-80d4-51a89956dca3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:41.457547 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.457395 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-config" (OuterVolumeSpecName: "console-config") pod "54a6416d-fadc-4fd9-80d4-51a89956dca3" (UID: "54a6416d-fadc-4fd9-80d4-51a89956dca3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:41.459336 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.459302 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "54a6416d-fadc-4fd9-80d4-51a89956dca3" (UID: "54a6416d-fadc-4fd9-80d4-51a89956dca3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:00:41.459448 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.459399 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "54a6416d-fadc-4fd9-80d4-51a89956dca3" (UID: "54a6416d-fadc-4fd9-80d4-51a89956dca3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:00:41.459490 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.459467 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a6416d-fadc-4fd9-80d4-51a89956dca3-kube-api-access-jnqvs" (OuterVolumeSpecName: "kube-api-access-jnqvs") pod "54a6416d-fadc-4fd9-80d4-51a89956dca3" (UID: "54a6416d-fadc-4fd9-80d4-51a89956dca3"). InnerVolumeSpecName "kube-api-access-jnqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:00:41.558278 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.558223 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnqvs\" (UniqueName: \"kubernetes.io/projected/54a6416d-fadc-4fd9-80d4-51a89956dca3-kube-api-access-jnqvs\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:41.558278 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.558273 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-oauth-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:41.558278 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.558289 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:41.558524 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.558301 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:41.558524 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.558313 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-oauth-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:41.558524 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:41.558326 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54a6416d-fadc-4fd9-80d4-51a89956dca3-console-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:42.240932 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.240898 2565 generic.go:356] "Generic (PLEG): container finished" podID="54a6416d-fadc-4fd9-80d4-51a89956dca3" containerID="7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49" exitCode=0 Jul 07 22:00:42.241378 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.240944 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f7d455bf-fmsbb" Jul 07 22:00:42.241378 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.240970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f7d455bf-fmsbb" event={"ID":"54a6416d-fadc-4fd9-80d4-51a89956dca3","Type":"ContainerDied","Data":"7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49"} Jul 07 22:00:42.241378 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.241012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f7d455bf-fmsbb" event={"ID":"54a6416d-fadc-4fd9-80d4-51a89956dca3","Type":"ContainerDied","Data":"871949cd71c540d8a0c8da52ead0caab931912aa894708ed96cdf18282541be7"} Jul 07 22:00:42.241378 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.241066 2565 scope.go:117] "RemoveContainer" containerID="7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49" Jul 07 22:00:42.250708 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.250687 2565 scope.go:117] "RemoveContainer" containerID="7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49" Jul 07 22:00:42.251068 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:00:42.251041 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49\": container with ID starting with 7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49 not found: ID does not exist" containerID="7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49" Jul 07 22:00:42.251149 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.251082 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49"} err="failed to get container status \"7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49\": rpc error: code = NotFound desc = could not find container \"7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49\": container with ID starting with 7957b276e0e9d7826fc74fdbcafb729682161b12ef2f7ce32f6f6609087f4a49 not found: ID does not exist" Jul 07 22:00:42.265820 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.265786 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f7d455bf-fmsbb"] Jul 07 22:00:42.269706 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.269665 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8f7d455bf-fmsbb"] Jul 07 22:00:42.575459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:42.575424 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54a6416d-fadc-4fd9-80d4-51a89956dca3" path="/var/lib/kubelet/pods/54a6416d-fadc-4fd9-80d4-51a89956dca3/volumes" Jul 07 22:00:45.963067 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:45.963002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:46.041304 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:46.041272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:48.141116 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.140894 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-54fd84ccf6-cv2vg" podUID="88413977-a2b3-4a70-b159-84da1558c8aa" containerName="console" containerID="cri-o://2c92a7168afff549984311bbdd5b41151a2749fc9b9401e035bb8170d17655e1" gracePeriod=15 Jul 07 22:00:48.276303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.276260 2565 generic.go:356] "Generic (PLEG): container finished" podID="88413977-a2b3-4a70-b159-84da1558c8aa" containerID="2c92a7168afff549984311bbdd5b41151a2749fc9b9401e035bb8170d17655e1" exitCode=0 Jul 07 22:00:48.276471 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.276336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fd84ccf6-cv2vg" event={"ID":"88413977-a2b3-4a70-b159-84da1558c8aa","Type":"ContainerDied","Data":"2c92a7168afff549984311bbdd5b41151a2749fc9b9401e035bb8170d17655e1"} Jul 07 22:00:48.420928 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.420901 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:48.525396 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-trusted-ca-bundle\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525570 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8f54\" (UniqueName: \"kubernetes.io/projected/88413977-a2b3-4a70-b159-84da1558c8aa-kube-api-access-s8f54\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525570 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-oauth-serving-cert\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525570 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525510 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-oauth-config\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525767 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525640 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-console-config\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525767 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525687 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-serving-cert\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525767 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525700 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:48.525767 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525715 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-service-ca\") pod \"88413977-a2b3-4a70-b159-84da1558c8aa\" (UID: \"88413977-a2b3-4a70-b159-84da1558c8aa\") " Jul 07 22:00:48.525965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.525887 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:48.526090 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.526061 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:48.526270 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.526147 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-trusted-ca-bundle\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:48.526354 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.526288 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-oauth-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:48.526354 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.526070 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-console-config" (OuterVolumeSpecName: "console-config") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:48.527981 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.527949 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:00:48.528122 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.527952 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88413977-a2b3-4a70-b159-84da1558c8aa-kube-api-access-s8f54" (OuterVolumeSpecName: "kube-api-access-s8f54") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "kube-api-access-s8f54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:00:48.528122 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.528056 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88413977-a2b3-4a70-b159-84da1558c8aa" (UID: "88413977-a2b3-4a70-b159-84da1558c8aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:00:48.626948 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.626912 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-console-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:48.626948 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.626943 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:48.626948 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.626954 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88413977-a2b3-4a70-b159-84da1558c8aa-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:48.627197 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.626963 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f54\" (UniqueName: \"kubernetes.io/projected/88413977-a2b3-4a70-b159-84da1558c8aa-kube-api-access-s8f54\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:48.627197 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:48.626972 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88413977-a2b3-4a70-b159-84da1558c8aa-console-oauth-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:49.087670 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:49.087636 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qf59h" Jul 07 22:00:49.283612 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:49.283587 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fd84ccf6-cv2vg" Jul 07 22:00:49.283612 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:49.283597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fd84ccf6-cv2vg" event={"ID":"88413977-a2b3-4a70-b159-84da1558c8aa","Type":"ContainerDied","Data":"247dba65a299c7de49cfd83f5610898c8986badd835c51698c7da44b9272ae3d"} Jul 07 22:00:49.284234 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:49.283657 2565 scope.go:117] "RemoveContainer" containerID="2c92a7168afff549984311bbdd5b41151a2749fc9b9401e035bb8170d17655e1" Jul 07 22:00:49.300468 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:49.300435 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fd84ccf6-cv2vg"] Jul 07 22:00:49.308472 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:49.308437 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54fd84ccf6-cv2vg"] Jul 07 22:00:50.576601 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:50.576564 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88413977-a2b3-4a70-b159-84da1558c8aa" path="/var/lib/kubelet/pods/88413977-a2b3-4a70-b159-84da1558c8aa/volumes" Jul 07 22:00:52.098433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:52.098389 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:52.188163 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:52.188098 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:00:56.287943 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:56.287913 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:56.291857 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:56.291832 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78987745dc-qb8sq" Jul 07 22:00:59.280864 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.280796 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7dbc45b887-j7j88" podUID="ac5f117b-d131-47f0-8057-db5b823981d8" containerName="console" containerID="cri-o://08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120" gracePeriod=15 Jul 07 22:00:59.533655 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.533586 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:00:59.638146 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-oauth-config\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638165 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-trusted-ca-bundle\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-service-ca\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638226 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-console-config\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638281 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-serving-cert\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638524 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-oauth-serving-cert\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638524 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f772p\" (UniqueName: \"kubernetes.io/projected/ac5f117b-d131-47f0-8057-db5b823981d8-kube-api-access-f772p\") pod \"ac5f117b-d131-47f0-8057-db5b823981d8\" (UID: \"ac5f117b-d131-47f0-8057-db5b823981d8\") " Jul 07 22:00:59.638663 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638638 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:59.638731 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638653 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:59.638811 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638764 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:59.638949 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.638927 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-console-config" (OuterVolumeSpecName: "console-config") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:00:59.640651 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.640613 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5f117b-d131-47f0-8057-db5b823981d8-kube-api-access-f772p" (OuterVolumeSpecName: "kube-api-access-f772p") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "kube-api-access-f772p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:00:59.640764 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.640667 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:00:59.640808 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.640756 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac5f117b-d131-47f0-8057-db5b823981d8" (UID: "ac5f117b-d131-47f0-8057-db5b823981d8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:00:59.739310 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739271 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-oauth-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:59.739310 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739301 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f772p\" (UniqueName: \"kubernetes.io/projected/ac5f117b-d131-47f0-8057-db5b823981d8-kube-api-access-f772p\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:59.739310 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739311 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-oauth-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:59.739310 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739319 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-trusted-ca-bundle\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:59.739561 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739329 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:59.739561 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739338 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac5f117b-d131-47f0-8057-db5b823981d8-console-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:00:59.739561 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:00:59.739345 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5f117b-d131-47f0-8057-db5b823981d8-console-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:01:00.338409 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.338368 2565 generic.go:356] "Generic (PLEG): container finished" podID="ac5f117b-d131-47f0-8057-db5b823981d8" containerID="08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120" exitCode=0 Jul 07 22:01:00.338844 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.338421 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbc45b887-j7j88" Jul 07 22:01:00.338844 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.338453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc45b887-j7j88" event={"ID":"ac5f117b-d131-47f0-8057-db5b823981d8","Type":"ContainerDied","Data":"08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120"} Jul 07 22:01:00.338844 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.338505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbc45b887-j7j88" event={"ID":"ac5f117b-d131-47f0-8057-db5b823981d8","Type":"ContainerDied","Data":"b11917f62cccf6d15c01592f77784e9c12d89e32f96d9ae521b4f782da49e546"} Jul 07 22:01:00.338844 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.338527 2565 scope.go:117] "RemoveContainer" containerID="08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120" Jul 07 22:01:00.347664 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.347644 2565 scope.go:117] "RemoveContainer" containerID="08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120" Jul 07 22:01:00.347978 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:01:00.347961 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120\": container with ID starting with 08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120 not found: ID does not exist" containerID="08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120" Jul 07 22:01:00.348015 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.347988 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120"} err="failed to get container status \"08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120\": rpc error: code = NotFound desc = could not find container \"08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120\": container with ID starting with 08662efd37d47af87485ef391bd1d1fa55df1371e4c2d647e101400e0cebe120 not found: ID does not exist" Jul 07 22:01:00.361039 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.360984 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbc45b887-j7j88"] Jul 07 22:01:00.363739 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.363715 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dbc45b887-j7j88"] Jul 07 22:01:00.576384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:00.576347 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5f117b-d131-47f0-8057-db5b823981d8" path="/var/lib/kubelet/pods/ac5f117b-d131-47f0-8057-db5b823981d8/volumes" Jul 07 22:01:18.222529 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:18.222485 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:01:18.243869 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:18.243839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:01:18.437837 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:18.437809 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 22:01:27.542912 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:27.542837 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:27.610672 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:27.610635 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:38.070727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:38.070686 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:39.479214 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:39.479160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:39.543256 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:39.543222 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:46.406919 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.406885 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-587c4566f8-h6sjz"] Jul 07 22:01:46.407372 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407280 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88413977-a2b3-4a70-b159-84da1558c8aa" containerName="console" Jul 07 22:01:46.407372 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407293 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88413977-a2b3-4a70-b159-84da1558c8aa" containerName="console" Jul 07 22:01:46.407372 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407303 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54a6416d-fadc-4fd9-80d4-51a89956dca3" containerName="console" Jul 07 22:01:46.407372 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407311 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54a6416d-fadc-4fd9-80d4-51a89956dca3" containerName="console" Jul 07 22:01:46.407372 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407323 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5f117b-d131-47f0-8057-db5b823981d8" containerName="console" Jul 07 22:01:46.407372 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407329 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5f117b-d131-47f0-8057-db5b823981d8" containerName="console" Jul 07 22:01:46.407550 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407392 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88413977-a2b3-4a70-b159-84da1558c8aa" containerName="console" Jul 07 22:01:46.407550 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407398 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54a6416d-fadc-4fd9-80d4-51a89956dca3" containerName="console" Jul 07 22:01:46.407550 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.407404 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5f117b-d131-47f0-8057-db5b823981d8" containerName="console" Jul 07 22:01:46.410437 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.410417 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.419743 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.419715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587c4566f8-h6sjz"] Jul 07 22:01:46.471499 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.471464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:46.569267 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-serving-cert\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569436 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-console-config\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569436 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-oauth-config\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569436 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569375 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-trusted-ca-bundle\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569436 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-service-ca\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569563 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569469 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-oauth-serving-cert\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569563 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjs7r\" (UniqueName: \"kubernetes.io/projected/bf72611b-e625-43b3-80f8-7c217323d6c8-kube-api-access-bjs7r\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.569563 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.569491 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:46.670889 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-trusted-ca-bundle\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.670889 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-service-ca\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.670889 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-oauth-serving-cert\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671218 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjs7r\" (UniqueName: \"kubernetes.io/projected/bf72611b-e625-43b3-80f8-7c217323d6c8-kube-api-access-bjs7r\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671218 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-serving-cert\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671218 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-console-config\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671218 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.670995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-oauth-config\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671691 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.671668 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-service-ca\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671815 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.671791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-oauth-serving-cert\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671916 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.671899 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-console-config\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.671952 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.671931 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-trusted-ca-bundle\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.674131 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.674106 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-oauth-config\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.674256 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.674228 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-serving-cert\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.679729 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.679708 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjs7r\" (UniqueName: \"kubernetes.io/projected/bf72611b-e625-43b3-80f8-7c217323d6c8-kube-api-access-bjs7r\") pod \"console-587c4566f8-h6sjz\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.720843 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.720795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:46.858504 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:46.858466 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-587c4566f8-h6sjz"] Jul 07 22:01:46.863320 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:01:46.863291 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf72611b_e625_43b3_80f8_7c217323d6c8.slice/crio-cf879f84cbfe1a6c72cfc9bfcaf64e0a334b773231b171b824bdd82dcc936af5 WatchSource:0}: Error finding container cf879f84cbfe1a6c72cfc9bfcaf64e0a334b773231b171b824bdd82dcc936af5: Status 404 returned error can't find the container with id cf879f84cbfe1a6c72cfc9bfcaf64e0a334b773231b171b824bdd82dcc936af5 Jul 07 22:01:47.545985 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:47.545946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587c4566f8-h6sjz" event={"ID":"bf72611b-e625-43b3-80f8-7c217323d6c8","Type":"ContainerStarted","Data":"db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305"} Jul 07 22:01:47.546426 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:47.546004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587c4566f8-h6sjz" event={"ID":"bf72611b-e625-43b3-80f8-7c217323d6c8","Type":"ContainerStarted","Data":"cf879f84cbfe1a6c72cfc9bfcaf64e0a334b773231b171b824bdd82dcc936af5"} Jul 07 22:01:52.101329 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:52.101279 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:52.202843 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:52.202813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:56.721798 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:56.721699 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:56.721798 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:56.721767 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:56.726554 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:56.726519 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:56.747493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:56.747437 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-587c4566f8-h6sjz" podStartSLOduration=10.747420256 podStartE2EDuration="10.747420256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:01:47.574862705 +0000 UTC m=+163.566435933" watchObservedRunningTime="2026-07-07 22:01:56.747420256 +0000 UTC m=+172.738993510" Jul 07 22:01:57.600054 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:57.600002 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:01:57.646466 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:57.646434 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd897c74c-rss2g"] Jul 07 22:01:57.787792 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:57.787746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:57.888355 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:57.888259 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:01:57.962708 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:01:57.962665 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:04.837757 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:04.837714 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:04.949386 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:04.949345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:07.155053 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:07.154997 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:22.669192 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:22.669123 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-dd897c74c-rss2g" podUID="2549ac44-daf5-47b3-942d-8efcce6d82aa" containerName="console" containerID="cri-o://8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39" gracePeriod=15 Jul 07 22:02:22.921148 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:22.921082 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:02:23.004824 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.004785 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-trusted-ca-bundle\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005043 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.004868 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-oauth-config\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005043 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.004895 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c542p\" (UniqueName: \"kubernetes.io/projected/2549ac44-daf5-47b3-942d-8efcce6d82aa-kube-api-access-c542p\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005043 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.004923 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-service-ca\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005071 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-serving-cert\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-oauth-serving-cert\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005148 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-config\") pod \"2549ac44-daf5-47b3-942d-8efcce6d82aa\" (UID: \"2549ac44-daf5-47b3-942d-8efcce6d82aa\") " Jul 07 22:02:23.005429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005287 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:02:23.005429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005357 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:02:23.005630 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005607 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-trusted-ca-bundle\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.005703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005624 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-config" (OuterVolumeSpecName: "console-config") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:02:23.005703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005636 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.005703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.005633 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:02:23.007344 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.007320 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:02:23.007741 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.007720 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:02:23.007844 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.007778 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2549ac44-daf5-47b3-942d-8efcce6d82aa-kube-api-access-c542p" (OuterVolumeSpecName: "kube-api-access-c542p") pod "2549ac44-daf5-47b3-942d-8efcce6d82aa" (UID: "2549ac44-daf5-47b3-942d-8efcce6d82aa"). InnerVolumeSpecName "kube-api-access-c542p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:02:23.106790 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.106749 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c542p\" (UniqueName: \"kubernetes.io/projected/2549ac44-daf5-47b3-942d-8efcce6d82aa-kube-api-access-c542p\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.106790 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.106783 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.106790 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.106796 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-oauth-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.107068 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.106809 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.107068 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.106821 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549ac44-daf5-47b3-942d-8efcce6d82aa-console-oauth-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:02:23.703661 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.703622 2565 generic.go:356] "Generic (PLEG): container finished" podID="2549ac44-daf5-47b3-942d-8efcce6d82aa" containerID="8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39" exitCode=0 Jul 07 22:02:23.703661 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.703674 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd897c74c-rss2g" Jul 07 22:02:23.704186 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.703695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd897c74c-rss2g" event={"ID":"2549ac44-daf5-47b3-942d-8efcce6d82aa","Type":"ContainerDied","Data":"8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39"} Jul 07 22:02:23.704186 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.703745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd897c74c-rss2g" event={"ID":"2549ac44-daf5-47b3-942d-8efcce6d82aa","Type":"ContainerDied","Data":"a3e50d6548171bb1d1b462a24470d95be49084bccf1f565ee29d6a9244fd42ba"} Jul 07 22:02:23.704186 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.703773 2565 scope.go:117] "RemoveContainer" containerID="8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39" Jul 07 22:02:23.713132 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.713113 2565 scope.go:117] "RemoveContainer" containerID="8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39" Jul 07 22:02:23.713480 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:02:23.713461 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39\": container with ID starting with 8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39 not found: ID does not exist" containerID="8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39" Jul 07 22:02:23.713530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.713491 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39"} err="failed to get container status \"8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39\": rpc error: code = NotFound desc = could not find container \"8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39\": container with ID starting with 8d067e37b49777170217930625d4266df464ec06ba907b6fbe4c0121c0520f39 not found: ID does not exist" Jul 07 22:02:23.726805 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.726772 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd897c74c-rss2g"] Jul 07 22:02:23.737289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:23.737249 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dd897c74c-rss2g"] Jul 07 22:02:24.575811 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:24.575771 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2549ac44-daf5-47b3-942d-8efcce6d82aa" path="/var/lib/kubelet/pods/2549ac44-daf5-47b3-942d-8efcce6d82aa/volumes" Jul 07 22:02:30.598816 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:30.598777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:30.697961 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:30.697927 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:40.947377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:40.947339 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:52.571505 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:52.571465 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:53.419882 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:53.419844 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:56.182501 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:56.182469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:56.243396 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:56.243359 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:57.804106 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:57.804068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:57.921668 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:57.921628 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:58.262863 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:58.262831 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:58.394548 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:58.394513 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:02:59.228053 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.227981 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5qkqd"] Jul 07 22:02:59.228553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.228527 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2549ac44-daf5-47b3-942d-8efcce6d82aa" containerName="console" Jul 07 22:02:59.228630 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.228558 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2549ac44-daf5-47b3-942d-8efcce6d82aa" containerName="console" Jul 07 22:02:59.228680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.228661 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2549ac44-daf5-47b3-942d-8efcce6d82aa" containerName="console" Jul 07 22:02:59.230764 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.230741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.232863 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.232842 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 22:02:59.233400 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.233387 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5xqv7\"" Jul 07 22:02:59.233454 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.233436 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 22:02:59.240387 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.240358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5qkqd"] Jul 07 22:02:59.337218 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.337179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/412eb115-8844-4c76-918a-ce213ce98507-kube-api-access-gmr5h\") pod \"cert-manager-webhook-6486ffd997-5qkqd\" (UID: \"412eb115-8844-4c76-918a-ce213ce98507\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.337426 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.337228 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/412eb115-8844-4c76-918a-ce213ce98507-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5qkqd\" (UID: \"412eb115-8844-4c76-918a-ce213ce98507\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.438368 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.438333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/412eb115-8844-4c76-918a-ce213ce98507-kube-api-access-gmr5h\") pod \"cert-manager-webhook-6486ffd997-5qkqd\" (UID: \"412eb115-8844-4c76-918a-ce213ce98507\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.438368 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.438371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/412eb115-8844-4c76-918a-ce213ce98507-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5qkqd\" (UID: \"412eb115-8844-4c76-918a-ce213ce98507\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.453206 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.453164 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/412eb115-8844-4c76-918a-ce213ce98507-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5qkqd\" (UID: \"412eb115-8844-4c76-918a-ce213ce98507\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.453363 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.453232 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr5h\" (UniqueName: \"kubernetes.io/projected/412eb115-8844-4c76-918a-ce213ce98507-kube-api-access-gmr5h\") pod \"cert-manager-webhook-6486ffd997-5qkqd\" (UID: \"412eb115-8844-4c76-918a-ce213ce98507\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.545681 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.545638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:02:59.684251 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.684222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5qkqd"] Jul 07 22:02:59.687051 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:02:59.687003 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod412eb115_8844_4c76_918a_ce213ce98507.slice/crio-faa9a35095c9f089c11b1e5dee6bc729197419f39c736fc151b446026d4db69e WatchSource:0}: Error finding container faa9a35095c9f089c11b1e5dee6bc729197419f39c736fc151b446026d4db69e: Status 404 returned error can't find the container with id faa9a35095c9f089c11b1e5dee6bc729197419f39c736fc151b446026d4db69e Jul 07 22:02:59.853096 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:02:59.852992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" event={"ID":"412eb115-8844-4c76-918a-ce213ce98507","Type":"ContainerStarted","Data":"faa9a35095c9f089c11b1e5dee6bc729197419f39c736fc151b446026d4db69e"} Jul 07 22:03:00.700634 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:00.700596 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:00.835123 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:00.835066 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:02.987730 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:02.987692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:03.088283 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:03.088254 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:03.880099 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:03.880060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" event={"ID":"412eb115-8844-4c76-918a-ce213ce98507","Type":"ContainerStarted","Data":"5ae08abfc85f3f9455730972350509faf0a5680d6e64fb3a0c63cc42c21b4481"} Jul 07 22:03:03.880302 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:03.880187 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:03:03.899301 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:03.899229 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" podStartSLOduration=1.6898789239999998 podStartE2EDuration="4.899206338s" totalImagesPullingTime="3.209327414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:03.896843757 +0000 UTC m=+239.888416987" watchObservedRunningTime="2026-07-07 22:03:03.899206338 +0000 UTC m=+239.890779569" Jul 07 22:03:05.865377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:05.865341 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:08.235859 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:08.235813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:09.204884 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.204846 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s2s4l"] Jul 07 22:03:09.208225 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.208205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.210340 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.210320 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dz2vr\"" Jul 07 22:03:09.219523 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.219486 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s2s4l"] Jul 07 22:03:09.335158 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.335104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44aa0c5d-6e4d-4be7-939a-5291c2bff3e6-bound-sa-token\") pod \"cert-manager-55b4d954d5-s2s4l\" (UID: \"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6\") " pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.335567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.335282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnk9\" (UniqueName: \"kubernetes.io/projected/44aa0c5d-6e4d-4be7-939a-5291c2bff3e6-kube-api-access-tqnk9\") pod \"cert-manager-55b4d954d5-s2s4l\" (UID: \"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6\") " pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.436086 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.436051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnk9\" (UniqueName: \"kubernetes.io/projected/44aa0c5d-6e4d-4be7-939a-5291c2bff3e6-kube-api-access-tqnk9\") pod \"cert-manager-55b4d954d5-s2s4l\" (UID: \"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6\") " pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.436252 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.436100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44aa0c5d-6e4d-4be7-939a-5291c2bff3e6-bound-sa-token\") pod \"cert-manager-55b4d954d5-s2s4l\" (UID: \"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6\") " pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.446176 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.446144 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44aa0c5d-6e4d-4be7-939a-5291c2bff3e6-bound-sa-token\") pod \"cert-manager-55b4d954d5-s2s4l\" (UID: \"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6\") " pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.446334 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.446241 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnk9\" (UniqueName: \"kubernetes.io/projected/44aa0c5d-6e4d-4be7-939a-5291c2bff3e6-kube-api-access-tqnk9\") pod \"cert-manager-55b4d954d5-s2s4l\" (UID: \"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6\") " pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.517778 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.517739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s2s4l" Jul 07 22:03:09.654919 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.653212 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s2s4l"] Jul 07 22:03:09.656780 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:03:09.656748 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44aa0c5d_6e4d_4be7_939a_5291c2bff3e6.slice/crio-904d3d72b1cb178f48a6d97ddccacb4b38aec080caff607476f555a3d643068a WatchSource:0}: Error finding container 904d3d72b1cb178f48a6d97ddccacb4b38aec080caff607476f555a3d643068a: Status 404 returned error can't find the container with id 904d3d72b1cb178f48a6d97ddccacb4b38aec080caff607476f555a3d643068a Jul 07 22:03:09.886714 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.886632 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5qkqd" Jul 07 22:03:09.907325 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.907286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s2s4l" event={"ID":"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6","Type":"ContainerStarted","Data":"b45e86e1f11bb27b843854899629d49cd77dbd3baea2120c36d96a4fadc5f58b"} Jul 07 22:03:09.907325 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.907330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s2s4l" event={"ID":"44aa0c5d-6e4d-4be7-939a-5291c2bff3e6","Type":"ContainerStarted","Data":"904d3d72b1cb178f48a6d97ddccacb4b38aec080caff607476f555a3d643068a"} Jul 07 22:03:09.925764 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:09.925714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-s2s4l" podStartSLOduration=0.925697336 podStartE2EDuration="925.697336ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:09.924258549 +0000 UTC m=+245.915831776" watchObservedRunningTime="2026-07-07 22:03:09.925697336 +0000 UTC m=+245.917270563" Jul 07 22:03:10.791991 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:10.791961 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:10.888582 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:10.888548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:16.356900 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:16.356868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:16.504479 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:16.504441 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:18.061588 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:18.061554 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:23.503998 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:23.503958 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:30.294158 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.294057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:30.426975 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.426929 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:30.482955 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.482920 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5"] Jul 07 22:03:30.486572 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.486547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.489574 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.489546 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 22:03:30.489779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.489583 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 22:03:30.489779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.489604 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 22:03:30.489779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.489542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mn624\"" Jul 07 22:03:30.489779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.489642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 22:03:30.489779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.489545 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 22:03:30.498930 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.498891 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5"] Jul 07 22:03:30.529279 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.529236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/913c53c1-f6bb-46c1-a02a-998e51577862-manager-config\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.529459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.529323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/913c53c1-f6bb-46c1-a02a-998e51577862-metrics-cert\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.529459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.529425 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/913c53c1-f6bb-46c1-a02a-998e51577862-cert\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.529459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.529442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs74t\" (UniqueName: \"kubernetes.io/projected/913c53c1-f6bb-46c1-a02a-998e51577862-kube-api-access-hs74t\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.598291 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.598181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:30.630074 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.630006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/913c53c1-f6bb-46c1-a02a-998e51577862-cert\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.630074 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.630076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs74t\" (UniqueName: \"kubernetes.io/projected/913c53c1-f6bb-46c1-a02a-998e51577862-kube-api-access-hs74t\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.630328 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.630103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/913c53c1-f6bb-46c1-a02a-998e51577862-manager-config\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.630328 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.630138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/913c53c1-f6bb-46c1-a02a-998e51577862-metrics-cert\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.630783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.630758 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/913c53c1-f6bb-46c1-a02a-998e51577862-manager-config\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.632615 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.632592 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/913c53c1-f6bb-46c1-a02a-998e51577862-metrics-cert\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.632716 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.632644 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/913c53c1-f6bb-46c1-a02a-998e51577862-cert\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.648858 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.648824 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs74t\" (UniqueName: \"kubernetes.io/projected/913c53c1-f6bb-46c1-a02a-998e51577862-kube-api-access-hs74t\") pod \"lws-controller-manager-5c4f8f67f4-l8gx5\" (UID: \"913c53c1-f6bb-46c1-a02a-998e51577862\") " pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.797148 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.797114 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:30.902009 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.901947 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:30.945049 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.945000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5"] Jul 07 22:03:30.947638 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:03:30.947612 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913c53c1_f6bb_46c1_a02a_998e51577862.slice/crio-c594d5df9080acf80976e6a5543fbc8d46d88495eb2351d0c8ebcda25e5acc05 WatchSource:0}: Error finding container c594d5df9080acf80976e6a5543fbc8d46d88495eb2351d0c8ebcda25e5acc05: Status 404 returned error can't find the container with id c594d5df9080acf80976e6a5543fbc8d46d88495eb2351d0c8ebcda25e5acc05 Jul 07 22:03:30.994365 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:30.994325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" event={"ID":"913c53c1-f6bb-46c1-a02a-998e51577862","Type":"ContainerStarted","Data":"c594d5df9080acf80976e6a5543fbc8d46d88495eb2351d0c8ebcda25e5acc05"} Jul 07 22:03:31.575415 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.575384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:31.691404 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.691367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:31.800295 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.800260 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw"] Jul 07 22:03:31.804655 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.804618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.809419 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.809177 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 22:03:31.809419 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.809254 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m6z9x\"" Jul 07 22:03:31.810507 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.810476 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 22:03:31.814978 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.814865 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 22:03:31.824890 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.824817 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 22:03:31.840428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.840335 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw"] Jul 07 22:03:31.845978 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.845942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae6f99ef-f667-4660-8361-d028816dc07f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.846175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.846048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae6f99ef-f667-4660-8361-d028816dc07f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.846175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.846086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae6f99ef-f667-4660-8361-d028816dc07f-tmp\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.846175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.846140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8lk\" (UniqueName: \"kubernetes.io/projected/ae6f99ef-f667-4660-8361-d028816dc07f-kube-api-access-2k8lk\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.846354 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.846222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae6f99ef-f667-4660-8361-d028816dc07f-manifests\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.947568 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.947464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae6f99ef-f667-4660-8361-d028816dc07f-manifests\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.947769 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.947611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae6f99ef-f667-4660-8361-d028816dc07f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.947769 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.947691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae6f99ef-f667-4660-8361-d028816dc07f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.947769 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.947730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae6f99ef-f667-4660-8361-d028816dc07f-tmp\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.947925 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.947789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8lk\" (UniqueName: \"kubernetes.io/projected/ae6f99ef-f667-4660-8361-d028816dc07f-kube-api-access-2k8lk\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.947925 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.947851 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae6f99ef-f667-4660-8361-d028816dc07f-manifests\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.951201 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.951171 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae6f99ef-f667-4660-8361-d028816dc07f-tmp\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.951785 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.951760 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae6f99ef-f667-4660-8361-d028816dc07f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.953818 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.953792 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae6f99ef-f667-4660-8361-d028816dc07f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:31.957843 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:31.957794 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8lk\" (UniqueName: \"kubernetes.io/projected/ae6f99ef-f667-4660-8361-d028816dc07f-kube-api-access-2k8lk\") pod \"opendatahub-operator-controller-manager-6dd55458bc-kj6zw\" (UID: \"ae6f99ef-f667-4660-8361-d028816dc07f\") " pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:32.029112 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.029071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:32.121522 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.121433 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:32.227949 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.227911 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:32.312860 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.312825 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw"] Jul 07 22:03:32.316425 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:03:32.316389 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6f99ef_f667_4660_8361_d028816dc07f.slice/crio-7921ffc39661675fd4fd2aaf84bdd25808b35b8b1850981ab2851ea8a7b90e4a WatchSource:0}: Error finding container 7921ffc39661675fd4fd2aaf84bdd25808b35b8b1850981ab2851ea8a7b90e4a: Status 404 returned error can't find the container with id 7921ffc39661675fd4fd2aaf84bdd25808b35b8b1850981ab2851ea8a7b90e4a Jul 07 22:03:32.323597 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.323563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:32.456721 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.456624 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:32.591906 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:32.591852 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:33.008483 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:33.008451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" event={"ID":"ae6f99ef-f667-4660-8361-d028816dc07f","Type":"ContainerStarted","Data":"7921ffc39661675fd4fd2aaf84bdd25808b35b8b1850981ab2851ea8a7b90e4a"} Jul 07 22:03:34.016946 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:34.016905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" event={"ID":"913c53c1-f6bb-46c1-a02a-998e51577862","Type":"ContainerStarted","Data":"7d17470e1ccda08e1378f11fd1421e198276b770cfc463968ba86585f8dc532e"} Jul 07 22:03:34.017398 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:34.017166 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:34.042420 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:34.042353 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" podStartSLOduration=1.485990779 podStartE2EDuration="4.042328397s" totalImagesPullingTime="2.556337618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:34.039877315 +0000 UTC m=+270.031450549" watchObservedRunningTime="2026-07-07 22:03:34.042328397 +0000 UTC m=+270.033901626" Jul 07 22:03:37.030959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:37.030922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" event={"ID":"ae6f99ef-f667-4660-8361-d028816dc07f","Type":"ContainerStarted","Data":"f33d545ddb072957e24a1e3f0ca81210371321a8a83f0550beb720add2a367cc"} Jul 07 22:03:38.037578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:38.037548 2565 generic.go:356] "Generic (PLEG): container finished" podID="ae6f99ef-f667-4660-8361-d028816dc07f" containerID="f33d545ddb072957e24a1e3f0ca81210371321a8a83f0550beb720add2a367cc" exitCode=0 Jul 07 22:03:38.037983 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:38.037637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" event={"ID":"ae6f99ef-f667-4660-8361-d028816dc07f","Type":"ContainerDied","Data":"f33d545ddb072957e24a1e3f0ca81210371321a8a83f0550beb720add2a367cc"} Jul 07 22:03:39.044839 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:39.044800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" event={"ID":"ae6f99ef-f667-4660-8361-d028816dc07f","Type":"ContainerStarted","Data":"d1d241e2b936a081a0b6c2fb8d5ff59344de3bd6b36337efaf93014a6d82ae3f"} Jul 07 22:03:39.045360 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:39.044905 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:39.071833 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:39.071764 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" podStartSLOduration=2.742046403 podStartE2EDuration="8.071741101s" totalImagesPullingTime="4.269298056s" totalInitContainerRuntime="1.060396642s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:39.068625302 +0000 UTC m=+275.060198531" watchObservedRunningTime="2026-07-07 22:03:39.071741101 +0000 UTC m=+275.063314329" Jul 07 22:03:44.978466 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:44.978432 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:45.023466 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:45.023436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c4f8f67f4-l8gx5" Jul 07 22:03:45.080107 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:45.080072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:50.052702 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:50.052671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dd55458bc-kj6zw" Jul 07 22:03:51.948578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:51.948542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:52.654520 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.654464 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf"] Jul 07 22:03:52.659980 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.659951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.665615 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.665568 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 22:03:52.666236 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.666212 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 22:03:52.666384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.666366 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 22:03:52.666483 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.666425 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 22:03:52.667009 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.666916 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l4kfg\"" Jul 07 22:03:52.671226 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.671202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf"] Jul 07 22:03:52.738675 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.738640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/099075b9-b650-496c-88be-c9c278c030ab-tmp\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.738848 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.738773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099075b9-b650-496c-88be-c9c278c030ab-tls-certs\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.738848 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.738823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjgd\" (UniqueName: \"kubernetes.io/projected/099075b9-b650-496c-88be-c9c278c030ab-kube-api-access-hcjgd\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.840324 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.840285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099075b9-b650-496c-88be-c9c278c030ab-tls-certs\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.840491 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.840343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjgd\" (UniqueName: \"kubernetes.io/projected/099075b9-b650-496c-88be-c9c278c030ab-kube-api-access-hcjgd\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.840491 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.840379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/099075b9-b650-496c-88be-c9c278c030ab-tmp\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.842754 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.842729 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/099075b9-b650-496c-88be-c9c278c030ab-tmp\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.842915 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.842893 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/099075b9-b650-496c-88be-c9c278c030ab-tls-certs\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.850527 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.850490 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjgd\" (UniqueName: \"kubernetes.io/projected/099075b9-b650-496c-88be-c9c278c030ab-kube-api-access-hcjgd\") pod \"kube-auth-proxy-98fbf9c8c-xh7bf\" (UID: \"099075b9-b650-496c-88be-c9c278c030ab\") " pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:52.970743 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:52.970652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" Jul 07 22:03:53.117120 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:53.117072 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf"] Jul 07 22:03:53.121389 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:03:53.121355 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099075b9_b650_496c_88be_c9c278c030ab.slice/crio-2bc70c12a97fff40feb75c235d2965469f5ab66a6c29575636210407482d9bd2 WatchSource:0}: Error finding container 2bc70c12a97fff40feb75c235d2965469f5ab66a6c29575636210407482d9bd2: Status 404 returned error can't find the container with id 2bc70c12a97fff40feb75c235d2965469f5ab66a6c29575636210407482d9bd2 Jul 07 22:03:53.345789 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:53.345751 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:53.786722 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:53.786659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:53.919958 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:53.919926 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:54.115282 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:54.115175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" event={"ID":"099075b9-b650-496c-88be-c9c278c030ab","Type":"ContainerStarted","Data":"2bc70c12a97fff40feb75c235d2965469f5ab66a6c29575636210407482d9bd2"} Jul 07 22:03:54.202598 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:54.202509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:54.328969 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:54.328936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:54.483802 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:54.483708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:55.597894 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:55.597853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:03:57.132786 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:57.132745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" event={"ID":"099075b9-b650-496c-88be-c9c278c030ab","Type":"ContainerStarted","Data":"f06d8c344f6d4646a58306fe5418ee756f107c8925f97bbbd52762eae665c47e"} Jul 07 22:03:57.152457 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:03:57.152397 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-98fbf9c8c-xh7bf" podStartSLOduration=1.6901587120000001 podStartE2EDuration="5.152378822s" totalImagesPullingTime="3.46222011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:03:57.150913575 +0000 UTC m=+293.142486805" watchObservedRunningTime="2026-07-07 22:03:57.152378822 +0000 UTC m=+293.143952049" Jul 07 22:04:00.434858 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:00.434817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:00.602734 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:00.602698 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:00.766225 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:00.766185 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:00.895806 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:00.895768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:01.078477 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:01.078390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:04.417960 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:04.417928 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 22:04:04.590531 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:04.590358 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:04.714363 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:04.714261 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:05.009618 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:05.009592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:05.914580 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:05.914537 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:06.059987 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:06.059955 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:06.188122 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:06.188008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:10.569039 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:10.569001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:10.660276 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:10.660244 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:20.059878 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:20.059841 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:20.160832 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:20.160798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:37.108204 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:37.108146 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:37.183927 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:37.183895 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:53.414199 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:53.414163 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:53.510345 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:53.510284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:53.603205 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:53.603165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:54.346211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:54.346175 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:04:54.546010 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:04:54.545970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:18.775910 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:18.775874 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:22.222097 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:22.222060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:22.892328 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:22.892296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:23.704198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:23.704166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:27.652255 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:27.652218 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:30.464876 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:30.464840 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:30.777481 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:30.777445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:32.962178 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:32.962143 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:33.081863 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:33.081821 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:35.508184 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:35.508146 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:38.045600 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:38.045563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:38.145277 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:38.145239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:38.278465 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:38.278392 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:38.364152 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:38.364057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:42.069665 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:42.069630 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:42.204663 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:42.204629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:44.835972 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:44.835933 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:44.906881 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:44.906847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:45.957474 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:45.957436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:46.232566 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:46.232476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:46.331486 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:46.331450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:46.434542 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:46.434501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:49.551175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:49.551143 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:49.657169 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:49.657134 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:56.253822 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.253790 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7"] Jul 07 22:05:56.256963 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.256945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:05:56.259435 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.259409 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fkbk8\"" Jul 07 22:05:56.259566 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.259420 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 22:05:56.259928 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.259912 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 22:05:56.268950 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.268917 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7"] Jul 07 22:05:56.345691 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.345655 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndsh\" (UniqueName: \"kubernetes.io/projected/005c655e-4c7f-4777-acb4-68e99c1d299e-kube-api-access-nndsh\") pod \"limitador-operator-controller-manager-859d7b78bd-6kpw7\" (UID: \"005c655e-4c7f-4777-acb4-68e99c1d299e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:05:56.446393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.446360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndsh\" (UniqueName: \"kubernetes.io/projected/005c655e-4c7f-4777-acb4-68e99c1d299e-kube-api-access-nndsh\") pod \"limitador-operator-controller-manager-859d7b78bd-6kpw7\" (UID: \"005c655e-4c7f-4777-acb4-68e99c1d299e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:05:56.459774 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.459749 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndsh\" (UniqueName: \"kubernetes.io/projected/005c655e-4c7f-4777-acb4-68e99c1d299e-kube-api-access-nndsh\") pod \"limitador-operator-controller-manager-859d7b78bd-6kpw7\" (UID: \"005c655e-4c7f-4777-acb4-68e99c1d299e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:05:56.568659 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.568555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:05:56.712214 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.712171 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7"] Jul 07 22:05:56.718985 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:05:56.718947 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005c655e_4c7f_4777_acb4_68e99c1d299e.slice/crio-9b79984bf00130e018145bc0c6a5d2168f3510d4e97565792c24eaf4cf72c663 WatchSource:0}: Error finding container 9b79984bf00130e018145bc0c6a5d2168f3510d4e97565792c24eaf4cf72c663: Status 404 returned error can't find the container with id 9b79984bf00130e018145bc0c6a5d2168f3510d4e97565792c24eaf4cf72c663 Jul 07 22:05:56.721531 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:56.721504 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:05:57.565174 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:57.565144 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:57.663898 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:57.663869 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:57.711179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:57.711143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" event={"ID":"005c655e-4c7f-4777-acb4-68e99c1d299e","Type":"ContainerStarted","Data":"9b79984bf00130e018145bc0c6a5d2168f3510d4e97565792c24eaf4cf72c663"} Jul 07 22:05:58.186296 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:58.186252 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:58.299096 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:58.299057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:05:59.722380 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:59.722338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" event={"ID":"005c655e-4c7f-4777-acb4-68e99c1d299e","Type":"ContainerStarted","Data":"f0ca9245c5d5d736f57f7bf8599338c05dc1fdb539dc1f3a7ac81428545f394a"} Jul 07 22:05:59.722828 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:59.722479 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:05:59.743738 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:05:59.743650 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" podStartSLOduration=1.306185832 podStartE2EDuration="3.743628018s" totalImagesPullingTime="2.437442186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:05:59.743537187 +0000 UTC m=+415.735110416" watchObservedRunningTime="2026-07-07 22:05:59.743628018 +0000 UTC m=+415.735201248" Jul 07 22:06:01.533890 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:01.533842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:01.636132 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:01.636085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:03.713836 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:03.713806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:03.782668 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:03.782630 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:08.483154 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:08.483121 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:08.576044 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:08.576005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:10.729887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:10.729852 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6kpw7" Jul 07 22:06:39.010572 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:39.010478 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:39.273467 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:39.273372 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:52.301274 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:52.301236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:52.415963 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:52.415920 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:52.480840 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:52.480807 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:53.126320 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.126283 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-gszgp"] Jul 07 22:06:53.129838 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.129815 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:06:53.134223 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.134197 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-42m78\"" Jul 07 22:06:53.137814 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.137792 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-gszgp"] Jul 07 22:06:53.159307 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.159272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqj2\" (UniqueName: \"kubernetes.io/projected/5944d8ca-97d5-4bb2-950d-d218cb02ccc1-kube-api-access-btqj2\") pod \"authorino-58cd955675-gszgp\" (UID: \"5944d8ca-97d5-4bb2-950d-d218cb02ccc1\") " pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:06:53.172661 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.172634 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:53.259976 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.259933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqj2\" (UniqueName: \"kubernetes.io/projected/5944d8ca-97d5-4bb2-950d-d218cb02ccc1-kube-api-access-btqj2\") pod \"authorino-58cd955675-gszgp\" (UID: \"5944d8ca-97d5-4bb2-950d-d218cb02ccc1\") " pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:06:53.268567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.268542 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqj2\" (UniqueName: \"kubernetes.io/projected/5944d8ca-97d5-4bb2-950d-d218cb02ccc1-kube-api-access-btqj2\") pod \"authorino-58cd955675-gszgp\" (UID: \"5944d8ca-97d5-4bb2-950d-d218cb02ccc1\") " pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:06:53.270918 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.270890 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:06:53.439889 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.439809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:06:53.574117 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.574092 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-gszgp"] Jul 07 22:06:53.576729 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:06:53.576698 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5944d8ca_97d5_4bb2_950d_d218cb02ccc1.slice/crio-3c965897b01e9fb060fb193f187da770ba0425b3e25f5fe21898a74ca925c77b WatchSource:0}: Error finding container 3c965897b01e9fb060fb193f187da770ba0425b3e25f5fe21898a74ca925c77b: Status 404 returned error can't find the container with id 3c965897b01e9fb060fb193f187da770ba0425b3e25f5fe21898a74ca925c77b Jul 07 22:06:53.976952 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:53.976915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gszgp" event={"ID":"5944d8ca-97d5-4bb2-950d-d218cb02ccc1","Type":"ContainerStarted","Data":"3c965897b01e9fb060fb193f187da770ba0425b3e25f5fe21898a74ca925c77b"} Jul 07 22:06:57.784599 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:57.784561 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-gszgp"] Jul 07 22:06:58.005517 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:58.005475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gszgp" event={"ID":"5944d8ca-97d5-4bb2-950d-d218cb02ccc1","Type":"ContainerStarted","Data":"16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2"} Jul 07 22:06:58.022224 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:58.022167 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-gszgp" podStartSLOduration=1.388898078 podStartE2EDuration="5.022150971s" totalImagesPullingTime="3.633252893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:06:58.020271577 +0000 UTC m=+474.011844857" watchObservedRunningTime="2026-07-07 22:06:58.022150971 +0000 UTC m=+474.013724199" Jul 07 22:06:59.010637 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:59.010571 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-gszgp" podUID="5944d8ca-97d5-4bb2-950d-d218cb02ccc1" containerName="authorino" containerID="cri-o://16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2" gracePeriod=30 Jul 07 22:06:59.266771 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:59.266700 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:06:59.320139 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:59.320096 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btqj2\" (UniqueName: \"kubernetes.io/projected/5944d8ca-97d5-4bb2-950d-d218cb02ccc1-kube-api-access-btqj2\") pod \"5944d8ca-97d5-4bb2-950d-d218cb02ccc1\" (UID: \"5944d8ca-97d5-4bb2-950d-d218cb02ccc1\") " Jul 07 22:06:59.322436 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:59.322408 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5944d8ca-97d5-4bb2-950d-d218cb02ccc1-kube-api-access-btqj2" (OuterVolumeSpecName: "kube-api-access-btqj2") pod "5944d8ca-97d5-4bb2-950d-d218cb02ccc1" (UID: "5944d8ca-97d5-4bb2-950d-d218cb02ccc1"). InnerVolumeSpecName "kube-api-access-btqj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:06:59.421384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:06:59.421328 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqj2\" (UniqueName: \"kubernetes.io/projected/5944d8ca-97d5-4bb2-950d-d218cb02ccc1-kube-api-access-btqj2\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:07:00.016955 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.016917 2565 generic.go:356] "Generic (PLEG): container finished" podID="5944d8ca-97d5-4bb2-950d-d218cb02ccc1" containerID="16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2" exitCode=0 Jul 07 22:07:00.017429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.016962 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gszgp" Jul 07 22:07:00.017429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.017010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gszgp" event={"ID":"5944d8ca-97d5-4bb2-950d-d218cb02ccc1","Type":"ContainerDied","Data":"16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2"} Jul 07 22:07:00.017429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.017072 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gszgp" event={"ID":"5944d8ca-97d5-4bb2-950d-d218cb02ccc1","Type":"ContainerDied","Data":"3c965897b01e9fb060fb193f187da770ba0425b3e25f5fe21898a74ca925c77b"} Jul 07 22:07:00.017429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.017101 2565 scope.go:117] "RemoveContainer" containerID="16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2" Jul 07 22:07:00.026469 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.026448 2565 scope.go:117] "RemoveContainer" containerID="16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2" Jul 07 22:07:00.026755 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:07:00.026736 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2\": container with ID starting with 16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2 not found: ID does not exist" containerID="16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2" Jul 07 22:07:00.026799 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.026767 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2"} err="failed to get container status \"16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2\": rpc error: code = NotFound desc = could not find container \"16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2\": container with ID starting with 16055315d83ac69f89bef610b70504ad7117e02af39b26268418a9e107fce0f2 not found: ID does not exist" Jul 07 22:07:00.038821 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.038794 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-gszgp"] Jul 07 22:07:00.042479 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.042455 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-gszgp"] Jul 07 22:07:00.576563 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:00.576532 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5944d8ca-97d5-4bb2-950d-d218cb02ccc1" path="/var/lib/kubelet/pods/5944d8ca-97d5-4bb2-950d-d218cb02ccc1/volumes" Jul 07 22:07:11.216188 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.216152 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:11.316349 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.316308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:11.659349 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.659317 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5"] Jul 07 22:07:11.659734 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.659716 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5944d8ca-97d5-4bb2-950d-d218cb02ccc1" containerName="authorino" Jul 07 22:07:11.659734 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.659731 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5944d8ca-97d5-4bb2-950d-d218cb02ccc1" containerName="authorino" Jul 07 22:07:11.659887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.659795 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5944d8ca-97d5-4bb2-950d-d218cb02ccc1" containerName="authorino" Jul 07 22:07:11.663912 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.663893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.666110 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.666085 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 07 22:07:11.666243 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.666084 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 07 22:07:11.666603 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.666587 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 07 22:07:11.666679 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.666591 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5jx4s\"" Jul 07 22:07:11.669752 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.669727 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5"] Jul 07 22:07:11.696491 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.696450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:11.734393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.734358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05c4fe39-1038-4ae0-bd11-264fb22235e8-data\") pod \"postgres-69fb4594cf-5zxj5\" (UID: \"05c4fe39-1038-4ae0-bd11-264fb22235e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.734570 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.734412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5z7\" (UniqueName: \"kubernetes.io/projected/05c4fe39-1038-4ae0-bd11-264fb22235e8-kube-api-access-qg5z7\") pod \"postgres-69fb4594cf-5zxj5\" (UID: \"05c4fe39-1038-4ae0-bd11-264fb22235e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.834977 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.834933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05c4fe39-1038-4ae0-bd11-264fb22235e8-data\") pod \"postgres-69fb4594cf-5zxj5\" (UID: \"05c4fe39-1038-4ae0-bd11-264fb22235e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.835172 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.834993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5z7\" (UniqueName: \"kubernetes.io/projected/05c4fe39-1038-4ae0-bd11-264fb22235e8-kube-api-access-qg5z7\") pod \"postgres-69fb4594cf-5zxj5\" (UID: \"05c4fe39-1038-4ae0-bd11-264fb22235e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.835423 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.835397 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05c4fe39-1038-4ae0-bd11-264fb22235e8-data\") pod \"postgres-69fb4594cf-5zxj5\" (UID: \"05c4fe39-1038-4ae0-bd11-264fb22235e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.844146 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.844119 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5z7\" (UniqueName: \"kubernetes.io/projected/05c4fe39-1038-4ae0-bd11-264fb22235e8-kube-api-access-qg5z7\") pod \"postgres-69fb4594cf-5zxj5\" (UID: \"05c4fe39-1038-4ae0-bd11-264fb22235e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:11.976820 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:11.976707 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:12.118215 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:12.118049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5"] Jul 07 22:07:12.121190 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:07:12.121161 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c4fe39_1038_4ae0_bd11_264fb22235e8.slice/crio-5919cf98a7163320306514da0b5839deb2ad1da5d4446c4871784f50fa825a6e WatchSource:0}: Error finding container 5919cf98a7163320306514da0b5839deb2ad1da5d4446c4871784f50fa825a6e: Status 404 returned error can't find the container with id 5919cf98a7163320306514da0b5839deb2ad1da5d4446c4871784f50fa825a6e Jul 07 22:07:13.084624 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:13.084577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" event={"ID":"05c4fe39-1038-4ae0-bd11-264fb22235e8","Type":"ContainerStarted","Data":"5919cf98a7163320306514da0b5839deb2ad1da5d4446c4871784f50fa825a6e"} Jul 07 22:07:15.988813 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:15.988761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:16.053358 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:16.053323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:18.112513 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:18.112465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" event={"ID":"05c4fe39-1038-4ae0-bd11-264fb22235e8","Type":"ContainerStarted","Data":"d3c79f29701c626d5d41568c112e0c4e7d71e662cfb481f9cfc3f437035f039a"} Jul 07 22:07:18.113067 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:18.112653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:18.135239 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:18.135175 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" podStartSLOduration=2.030149418 podStartE2EDuration="7.135157723s" totalImagesPullingTime="5.105008305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:07:18.133185203 +0000 UTC m=+494.124758433" watchObservedRunningTime="2026-07-07 22:07:18.135157723 +0000 UTC m=+494.126730950" Jul 07 22:07:24.147561 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:24.147527 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5zxj5" Jul 07 22:07:24.495309 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:24.495212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:24.595901 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:24.595868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:25.052201 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.052161 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c7c89cfdb-j4vrn"] Jul 07 22:07:25.131852 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.131811 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c7c89cfdb-j4vrn"] Jul 07 22:07:25.132016 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.131909 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.134703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.134681 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-42m78\"" Jul 07 22:07:25.134835 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.134710 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 22:07:25.267558 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.267521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/961f7fe6-11ba-4f43-82b0-42a71aa956e8-tls-cert\") pod \"authorino-c7c89cfdb-j4vrn\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.267975 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.267638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw65n\" (UniqueName: \"kubernetes.io/projected/961f7fe6-11ba-4f43-82b0-42a71aa956e8-kube-api-access-cw65n\") pod \"authorino-c7c89cfdb-j4vrn\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.369311 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.369221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/961f7fe6-11ba-4f43-82b0-42a71aa956e8-tls-cert\") pod \"authorino-c7c89cfdb-j4vrn\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.369311 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.369307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw65n\" (UniqueName: \"kubernetes.io/projected/961f7fe6-11ba-4f43-82b0-42a71aa956e8-kube-api-access-cw65n\") pod \"authorino-c7c89cfdb-j4vrn\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.372073 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.372016 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/961f7fe6-11ba-4f43-82b0-42a71aa956e8-tls-cert\") pod \"authorino-c7c89cfdb-j4vrn\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.378208 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.378181 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw65n\" (UniqueName: \"kubernetes.io/projected/961f7fe6-11ba-4f43-82b0-42a71aa956e8-kube-api-access-cw65n\") pod \"authorino-c7c89cfdb-j4vrn\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.441058 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.440982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:07:25.611585 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:25.611558 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c7c89cfdb-j4vrn"] Jul 07 22:07:25.613897 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:07:25.613857 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod961f7fe6_11ba_4f43_82b0_42a71aa956e8.slice/crio-313341952ec7afcd42a6f799d22752cb03f713a968a27eb36bc7c17c8b10b240 WatchSource:0}: Error finding container 313341952ec7afcd42a6f799d22752cb03f713a968a27eb36bc7c17c8b10b240: Status 404 returned error can't find the container with id 313341952ec7afcd42a6f799d22752cb03f713a968a27eb36bc7c17c8b10b240 Jul 07 22:07:26.150885 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:26.150849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" event={"ID":"961f7fe6-11ba-4f43-82b0-42a71aa956e8","Type":"ContainerStarted","Data":"313341952ec7afcd42a6f799d22752cb03f713a968a27eb36bc7c17c8b10b240"} Jul 07 22:07:27.158069 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:27.158003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" event={"ID":"961f7fe6-11ba-4f43-82b0-42a71aa956e8","Type":"ContainerStarted","Data":"d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5"} Jul 07 22:07:27.176587 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:27.176530 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" podStartSLOduration=1.705748861 podStartE2EDuration="2.176511046s" totalImagesPullingTime="470.762185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:07:27.174836867 +0000 UTC m=+503.166410095" watchObservedRunningTime="2026-07-07 22:07:27.176511046 +0000 UTC m=+503.168084273" Jul 07 22:07:31.717439 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:31.717403 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:31.817321 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:31.817282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:31.922389 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:31.922354 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:32.162793 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:32.162762 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:34.942569 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:34.942529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:35.061710 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:35.061668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:37.102217 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:37.102184 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:07:37.201695 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:07:37.201659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:06.345413 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:06.345333 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:06.835648 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:06.835610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:07.045340 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:07.045304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:07.454127 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:07.454091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:07.551773 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:07.551718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:17.742150 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:17.742113 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:17.842018 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:17.841983 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:23.067919 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:23.067880 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:23.169374 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:23.169338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:29.442395 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:29.442363 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:08:49.124137 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:08:49.124104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:09:09.775011 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.774969 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-26r48"] Jul 07 22:09:09.778382 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.778364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.780730 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.780699 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 22:09:09.781359 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.781338 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vjtr8\"" Jul 07 22:09:09.781474 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.781338 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 22:09:09.781474 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.781382 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 22:09:09.789957 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.789931 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-26r48"] Jul 07 22:09:09.816826 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.816790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:09:09.865905 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.865865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.865905 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.865909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.866220 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.865933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4df6b\" (UniqueName: \"kubernetes.io/projected/8249493d-51c6-4300-a89d-1566194ad24a-kube-api-access-4df6b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.866220 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.866058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.866220 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.866096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.866220 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.866129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249493d-51c6-4300-a89d-1566194ad24a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.966846 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.966805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.966846 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.966847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967155 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.966870 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4df6b\" (UniqueName: \"kubernetes.io/projected/8249493d-51c6-4300-a89d-1566194ad24a-kube-api-access-4df6b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967155 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.966896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967155 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.966914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967155 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.966930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249493d-51c6-4300-a89d-1566194ad24a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967362 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.967309 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967400 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.967365 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.967400 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.967388 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.969438 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.969418 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249493d-51c6-4300-a89d-1566194ad24a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.969586 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.969569 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249493d-51c6-4300-a89d-1566194ad24a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:09.975951 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:09.975920 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4df6b\" (UniqueName: \"kubernetes.io/projected/8249493d-51c6-4300-a89d-1566194ad24a-kube-api-access-4df6b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-26r48\" (UID: \"8249493d-51c6-4300-a89d-1566194ad24a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:10.090900 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:10.090813 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:10.242246 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:10.242206 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-26r48"] Jul 07 22:09:10.245999 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:09:10.245971 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8249493d_51c6_4300_a89d_1566194ad24a.slice/crio-c6a8a1a1cf8e217f1f87a518797dcfb0c248f3f3fe573aed8cc6631a3d1b9131 WatchSource:0}: Error finding container c6a8a1a1cf8e217f1f87a518797dcfb0c248f3f3fe573aed8cc6631a3d1b9131: Status 404 returned error can't find the container with id c6a8a1a1cf8e217f1f87a518797dcfb0c248f3f3fe573aed8cc6631a3d1b9131 Jul 07 22:09:10.656736 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:10.656695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" event={"ID":"8249493d-51c6-4300-a89d-1566194ad24a","Type":"ContainerStarted","Data":"c6a8a1a1cf8e217f1f87a518797dcfb0c248f3f3fe573aed8cc6631a3d1b9131"} Jul 07 22:09:11.665577 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:11.665475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" event={"ID":"8249493d-51c6-4300-a89d-1566194ad24a","Type":"ContainerStarted","Data":"7ad4862a3e0e0a0a56575bcfbee3fe997149f2af0d7beccbe524a9fd1301f7c0"} Jul 07 22:09:11.687268 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:11.687210 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" podStartSLOduration=1.6045598669999999 podStartE2EDuration="2.687193229s" totalImagesPullingTime="1.082633362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:09:11.685060898 +0000 UTC m=+607.676634127" watchObservedRunningTime="2026-07-07 22:09:11.687193229 +0000 UTC m=+607.678766457" Jul 07 22:09:18.869340 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:18.869308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:09:20.091198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:20.091159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:20.091652 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:20.091310 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:20.103980 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:20.103951 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:20.720611 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:20.720583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-26r48" Jul 07 22:09:25.230646 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:25.230553 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:09:32.896104 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:32.896057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:09:49.779874 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:49.779837 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:09:49.843616 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:09:49.843580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:36.701014 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.700971 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-779fd78f6f-txpr5"] Jul 07 22:10:36.704894 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.704871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.718715 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.718682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779fd78f6f-txpr5"] Jul 07 22:10:36.761804 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.761755 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:36.777549 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e008ede-34a9-462e-88d1-41575e11f1a5-console-oauth-config\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.777685 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-console-config\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.777685 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvg2\" (UniqueName: \"kubernetes.io/projected/6e008ede-34a9-462e-88d1-41575e11f1a5-kube-api-access-jgvg2\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.777685 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-service-ca\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.777800 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-oauth-serving-cert\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.777800 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-trusted-ca-bundle\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.777800 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.777764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e008ede-34a9-462e-88d1-41575e11f1a5-console-serving-cert\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.823298 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.823262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:36.879154 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-oauth-serving-cert\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879154 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-trusted-ca-bundle\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e008ede-34a9-462e-88d1-41575e11f1a5-console-serving-cert\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e008ede-34a9-462e-88d1-41575e11f1a5-console-oauth-config\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-console-config\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvg2\" (UniqueName: \"kubernetes.io/projected/6e008ede-34a9-462e-88d1-41575e11f1a5-kube-api-access-jgvg2\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-service-ca\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.879992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.879969 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-oauth-serving-cert\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.880126 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.880068 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-console-config\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.880126 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.880068 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-service-ca\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.880417 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.880396 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e008ede-34a9-462e-88d1-41575e11f1a5-trusted-ca-bundle\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.882013 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.881978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e008ede-34a9-462e-88d1-41575e11f1a5-console-oauth-config\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.882136 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.882066 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e008ede-34a9-462e-88d1-41575e11f1a5-console-serving-cert\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.887501 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.887477 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvg2\" (UniqueName: \"kubernetes.io/projected/6e008ede-34a9-462e-88d1-41575e11f1a5-kube-api-access-jgvg2\") pod \"console-779fd78f6f-txpr5\" (UID: \"6e008ede-34a9-462e-88d1-41575e11f1a5\") " pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:36.891217 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.891187 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:36.950347 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:36.950309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:37.015494 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:37.015449 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:37.163329 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:37.163288 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779fd78f6f-txpr5"] Jul 07 22:10:37.165384 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:10:37.165360 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e008ede_34a9_462e_88d1_41575e11f1a5.slice/crio-c4dd588c704eb480930a270e55437f3d343755cbaed6265ecd450b46e8eb66a5 WatchSource:0}: Error finding container c4dd588c704eb480930a270e55437f3d343755cbaed6265ecd450b46e8eb66a5: Status 404 returned error can't find the container with id c4dd588c704eb480930a270e55437f3d343755cbaed6265ecd450b46e8eb66a5 Jul 07 22:10:37.488903 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:37.488796 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:38.092569 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:38.092536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779fd78f6f-txpr5" event={"ID":"6e008ede-34a9-462e-88d1-41575e11f1a5","Type":"ContainerStarted","Data":"a95d567e715adabaf820ad524d4cbb9f0286812cc981e5a4e36e090280683b86"} Jul 07 22:10:38.093069 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:38.092590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779fd78f6f-txpr5" event={"ID":"6e008ede-34a9-462e-88d1-41575e11f1a5","Type":"ContainerStarted","Data":"c4dd588c704eb480930a270e55437f3d343755cbaed6265ecd450b46e8eb66a5"} Jul 07 22:10:38.110285 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:38.110221 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-779fd78f6f-txpr5" podStartSLOduration=2.110203589 podStartE2EDuration="2.110203589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:10:38.108152241 +0000 UTC m=+694.099725470" watchObservedRunningTime="2026-07-07 22:10:38.110203589 +0000 UTC m=+694.101776816" Jul 07 22:10:45.357288 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:45.357249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:45.416732 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:45.416698 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:46.162554 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.162508 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c7c89cfdb-j4vrn"] Jul 07 22:10:46.162771 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.162699 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" podUID="961f7fe6-11ba-4f43-82b0-42a71aa956e8" containerName="authorino" containerID="cri-o://d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5" gracePeriod=30 Jul 07 22:10:46.426977 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.426898 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:10:46.549153 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.549111 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:46.576141 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.576109 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw65n\" (UniqueName: \"kubernetes.io/projected/961f7fe6-11ba-4f43-82b0-42a71aa956e8-kube-api-access-cw65n\") pod \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " Jul 07 22:10:46.576309 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.576173 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/961f7fe6-11ba-4f43-82b0-42a71aa956e8-tls-cert\") pod \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\" (UID: \"961f7fe6-11ba-4f43-82b0-42a71aa956e8\") " Jul 07 22:10:46.578401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.578367 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961f7fe6-11ba-4f43-82b0-42a71aa956e8-kube-api-access-cw65n" (OuterVolumeSpecName: "kube-api-access-cw65n") pod "961f7fe6-11ba-4f43-82b0-42a71aa956e8" (UID: "961f7fe6-11ba-4f43-82b0-42a71aa956e8"). InnerVolumeSpecName "kube-api-access-cw65n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:10:46.588456 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.588416 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961f7fe6-11ba-4f43-82b0-42a71aa956e8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "961f7fe6-11ba-4f43-82b0-42a71aa956e8" (UID: "961f7fe6-11ba-4f43-82b0-42a71aa956e8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:10:46.647672 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.647637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:46.677262 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.677182 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw65n\" (UniqueName: \"kubernetes.io/projected/961f7fe6-11ba-4f43-82b0-42a71aa956e8-kube-api-access-cw65n\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:10:46.677262 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:46.677215 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/961f7fe6-11ba-4f43-82b0-42a71aa956e8-tls-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:10:47.015762 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.015722 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:47.015979 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.015829 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:47.020621 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.020592 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:47.146457 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.146420 2565 generic.go:356] "Generic (PLEG): container finished" podID="961f7fe6-11ba-4f43-82b0-42a71aa956e8" containerID="d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5" exitCode=0 Jul 07 22:10:47.146644 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.146473 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" Jul 07 22:10:47.146644 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.146509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" event={"ID":"961f7fe6-11ba-4f43-82b0-42a71aa956e8","Type":"ContainerDied","Data":"d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5"} Jul 07 22:10:47.146644 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.146558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c7c89cfdb-j4vrn" event={"ID":"961f7fe6-11ba-4f43-82b0-42a71aa956e8","Type":"ContainerDied","Data":"313341952ec7afcd42a6f799d22752cb03f713a968a27eb36bc7c17c8b10b240"} Jul 07 22:10:47.146644 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.146580 2565 scope.go:117] "RemoveContainer" containerID="d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5" Jul 07 22:10:47.151230 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.151208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-779fd78f6f-txpr5" Jul 07 22:10:47.156337 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.156319 2565 scope.go:117] "RemoveContainer" containerID="d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5" Jul 07 22:10:47.156654 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:10:47.156634 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5\": container with ID starting with d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5 not found: ID does not exist" containerID="d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5" Jul 07 22:10:47.156713 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.156663 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5"} err="failed to get container status \"d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5\": rpc error: code = NotFound desc = could not find container \"d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5\": container with ID starting with d906a3fd599d4d43db746a56ef536bf4d5f791befb522c6edb61e49d345c1da5 not found: ID does not exist" Jul 07 22:10:47.184502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.184412 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c7c89cfdb-j4vrn"] Jul 07 22:10:47.188182 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.188149 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c7c89cfdb-j4vrn"] Jul 07 22:10:47.197966 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.197933 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-587c4566f8-h6sjz"] Jul 07 22:10:47.327417 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.327322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:47.427578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.427542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:47.528322 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:47.528285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:10:48.576983 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:48.576945 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="961f7fe6-11ba-4f43-82b0-42a71aa956e8" path="/var/lib/kubelet/pods/961f7fe6-11ba-4f43-82b0-42a71aa956e8/volumes" Jul 07 22:10:54.691389 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:10:54.691301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:11:12.220855 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.220803 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-587c4566f8-h6sjz" podUID="bf72611b-e625-43b3-80f8-7c217323d6c8" containerName="console" containerID="cri-o://db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305" gracePeriod=15 Jul 07 22:11:12.481264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.481194 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:11:12.526179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526142 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-service-ca\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526188 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-trusted-ca-bundle\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526317 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-serving-cert\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526359 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjs7r\" (UniqueName: \"kubernetes.io/projected/bf72611b-e625-43b3-80f8-7c217323d6c8-kube-api-access-bjs7r\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526403 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-oauth-config\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-oauth-serving-cert\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-console-config\") pod \"bf72611b-e625-43b3-80f8-7c217323d6c8\" (UID: \"bf72611b-e625-43b3-80f8-7c217323d6c8\") " Jul 07 22:11:12.526741 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526597 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-service-ca" (OuterVolumeSpecName: "service-ca") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:11:12.526836 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526812 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:11:12.526893 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526838 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:11:12.526946 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526920 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-console-config" (OuterVolumeSpecName: "console-config") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:11:12.527010 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526941 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-oauth-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:12.527010 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526974 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:12.527010 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.526988 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-trusted-ca-bundle\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:12.528640 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.528614 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:11:12.529174 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.529150 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:11:12.529174 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.529162 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf72611b-e625-43b3-80f8-7c217323d6c8-kube-api-access-bjs7r" (OuterVolumeSpecName: "kube-api-access-bjs7r") pod "bf72611b-e625-43b3-80f8-7c217323d6c8" (UID: "bf72611b-e625-43b3-80f8-7c217323d6c8"). InnerVolumeSpecName "kube-api-access-bjs7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:11:12.627944 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.627909 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-oauth-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:12.627944 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.627939 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf72611b-e625-43b3-80f8-7c217323d6c8-console-config\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:12.627944 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.627949 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf72611b-e625-43b3-80f8-7c217323d6c8-console-serving-cert\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:12.628234 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:12.627958 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjs7r\" (UniqueName: \"kubernetes.io/projected/bf72611b-e625-43b3-80f8-7c217323d6c8-kube-api-access-bjs7r\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:11:13.274325 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.274289 2565 generic.go:356] "Generic (PLEG): container finished" podID="bf72611b-e625-43b3-80f8-7c217323d6c8" containerID="db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305" exitCode=0 Jul 07 22:11:13.274793 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.274341 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-587c4566f8-h6sjz" Jul 07 22:11:13.274793 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.274377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587c4566f8-h6sjz" event={"ID":"bf72611b-e625-43b3-80f8-7c217323d6c8","Type":"ContainerDied","Data":"db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305"} Jul 07 22:11:13.274793 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.274433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-587c4566f8-h6sjz" event={"ID":"bf72611b-e625-43b3-80f8-7c217323d6c8","Type":"ContainerDied","Data":"cf879f84cbfe1a6c72cfc9bfcaf64e0a334b773231b171b824bdd82dcc936af5"} Jul 07 22:11:13.274793 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.274454 2565 scope.go:117] "RemoveContainer" containerID="db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305" Jul 07 22:11:13.283343 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.283324 2565 scope.go:117] "RemoveContainer" containerID="db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305" Jul 07 22:11:13.283659 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:11:13.283636 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305\": container with ID starting with db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305 not found: ID does not exist" containerID="db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305" Jul 07 22:11:13.283709 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.283674 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305"} err="failed to get container status \"db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305\": rpc error: code = NotFound desc = could not find container \"db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305\": container with ID starting with db7282bc402b2d1fc6992d6930ba56cf482e76cd1c4ab346ce8eef647cbb4305 not found: ID does not exist" Jul 07 22:11:13.289795 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.289764 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-587c4566f8-h6sjz"] Jul 07 22:11:13.293285 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:13.293256 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-587c4566f8-h6sjz"] Jul 07 22:11:14.576329 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:14.576293 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf72611b-e625-43b3-80f8-7c217323d6c8" path="/var/lib/kubelet/pods/bf72611b-e625-43b3-80f8-7c217323d6c8/volumes" Jul 07 22:11:57.724495 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:57.724439 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:11:59.153958 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.153924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:11:59.269706 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.269667 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d68568969-cxrjg"] Jul 07 22:11:59.270086 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.270073 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf72611b-e625-43b3-80f8-7c217323d6c8" containerName="console" Jul 07 22:11:59.270137 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.270089 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf72611b-e625-43b3-80f8-7c217323d6c8" containerName="console" Jul 07 22:11:59.270137 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.270103 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="961f7fe6-11ba-4f43-82b0-42a71aa956e8" containerName="authorino" Jul 07 22:11:59.270137 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.270111 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="961f7fe6-11ba-4f43-82b0-42a71aa956e8" containerName="authorino" Jul 07 22:11:59.270232 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.270167 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="961f7fe6-11ba-4f43-82b0-42a71aa956e8" containerName="authorino" Jul 07 22:11:59.270232 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.270179 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf72611b-e625-43b3-80f8-7c217323d6c8" containerName="console" Jul 07 22:11:59.273356 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.273332 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.275422 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.275399 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 22:11:59.275889 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.275862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 22:11:59.275996 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.275918 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bdknk\"" Jul 07 22:11:59.283015 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.282988 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d68568969-cxrjg"] Jul 07 22:11:59.352423 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.352380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szv9v\" (UniqueName: \"kubernetes.io/projected/94c39d98-31c9-4bb1-b998-ff1a9be627f1-kube-api-access-szv9v\") pod \"maas-controller-d68568969-cxrjg\" (UID: \"94c39d98-31c9-4bb1-b998-ff1a9be627f1\") " pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.352606 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.352431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94c39d98-31c9-4bb1-b998-ff1a9be627f1-webhook-cert\") pod \"maas-controller-d68568969-cxrjg\" (UID: \"94c39d98-31c9-4bb1-b998-ff1a9be627f1\") " pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.453006 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.452907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szv9v\" (UniqueName: \"kubernetes.io/projected/94c39d98-31c9-4bb1-b998-ff1a9be627f1-kube-api-access-szv9v\") pod \"maas-controller-d68568969-cxrjg\" (UID: \"94c39d98-31c9-4bb1-b998-ff1a9be627f1\") " pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.453006 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.452966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94c39d98-31c9-4bb1-b998-ff1a9be627f1-webhook-cert\") pod \"maas-controller-d68568969-cxrjg\" (UID: \"94c39d98-31c9-4bb1-b998-ff1a9be627f1\") " pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.455592 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.455571 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94c39d98-31c9-4bb1-b998-ff1a9be627f1-webhook-cert\") pod \"maas-controller-d68568969-cxrjg\" (UID: \"94c39d98-31c9-4bb1-b998-ff1a9be627f1\") " pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.460951 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.460927 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szv9v\" (UniqueName: \"kubernetes.io/projected/94c39d98-31c9-4bb1-b998-ff1a9be627f1-kube-api-access-szv9v\") pod \"maas-controller-d68568969-cxrjg\" (UID: \"94c39d98-31c9-4bb1-b998-ff1a9be627f1\") " pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.584609 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.584572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:11:59.727803 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.727773 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d68568969-cxrjg"] Jul 07 22:11:59.729910 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:11:59.729883 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c39d98_31c9_4bb1_b998_ff1a9be627f1.slice/crio-7e34f8d5f9d0dc45892d96944377f08411fcc7cd8ad618fc6b5a4ca881246c3b WatchSource:0}: Error finding container 7e34f8d5f9d0dc45892d96944377f08411fcc7cd8ad618fc6b5a4ca881246c3b: Status 404 returned error can't find the container with id 7e34f8d5f9d0dc45892d96944377f08411fcc7cd8ad618fc6b5a4ca881246c3b Jul 07 22:11:59.731300 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:11:59.731277 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:12:00.511714 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:00.511676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d68568969-cxrjg" event={"ID":"94c39d98-31c9-4bb1-b998-ff1a9be627f1","Type":"ContainerStarted","Data":"7e34f8d5f9d0dc45892d96944377f08411fcc7cd8ad618fc6b5a4ca881246c3b"} Jul 07 22:12:02.524152 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:02.524098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d68568969-cxrjg" event={"ID":"94c39d98-31c9-4bb1-b998-ff1a9be627f1","Type":"ContainerStarted","Data":"5eb1d3fde5ada0b4335aa0f77cbe1f7a41afde7da01e499fe2cbbff0e82a29cc"} Jul 07 22:12:02.524558 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:02.524184 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:12:02.541668 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:02.541593 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d68568969-cxrjg" podStartSLOduration=1.525263423 podStartE2EDuration="3.541577873s" totalImagesPullingTime="2.01631445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:12:02.539014687 +0000 UTC m=+778.530587918" watchObservedRunningTime="2026-07-07 22:12:02.541577873 +0000 UTC m=+778.533151130" Jul 07 22:12:04.374903 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:04.374864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:04.424229 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:04.424189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:13.536293 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:13.536252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d68568969-cxrjg" Jul 07 22:12:44.144664 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:44.144579 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:44.245578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:44.245541 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:44.368085 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:44.368052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:50.200264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:50.200228 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:50.307473 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:50.307439 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:50.400282 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:50.400247 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:50.553240 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:50.553202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:56.216374 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:56.216336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:12:56.317644 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:12:56.317608 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:13:15.880550 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:13:15.880513 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:14:00.636583 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:14:00.636501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:14:00.740526 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:14:00.740494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:15:00.150507 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.150465 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f"] Jul 07 22:15:00.154103 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.154084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:15:00.156216 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.156192 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:15:00.171175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.171136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f"] Jul 07 22:15:00.261493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.261448 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx6v\" (UniqueName: \"kubernetes.io/projected/9225687e-e709-4c9e-908d-95d707eb4d39-kube-api-access-6kx6v\") pod \"maas-api-key-cleanup-29724375-55k2f\" (UID: \"9225687e-e709-4c9e-908d-95d707eb4d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:15:00.362194 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.362145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx6v\" (UniqueName: \"kubernetes.io/projected/9225687e-e709-4c9e-908d-95d707eb4d39-kube-api-access-6kx6v\") pod \"maas-api-key-cleanup-29724375-55k2f\" (UID: \"9225687e-e709-4c9e-908d-95d707eb4d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:15:00.372519 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.372476 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx6v\" (UniqueName: \"kubernetes.io/projected/9225687e-e709-4c9e-908d-95d707eb4d39-kube-api-access-6kx6v\") pod \"maas-api-key-cleanup-29724375-55k2f\" (UID: \"9225687e-e709-4c9e-908d-95d707eb4d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:15:00.472904 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.472803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:15:00.613911 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:00.613882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f"] Jul 07 22:15:00.616270 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:15:00.616237 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9225687e_e709_4c9e_908d_95d707eb4d39.slice/crio-c574457820be7f7bc29379d6d5e1cab51629810dfe2215bf4d37ed48aca829bd WatchSource:0}: Error finding container c574457820be7f7bc29379d6d5e1cab51629810dfe2215bf4d37ed48aca829bd: Status 404 returned error can't find the container with id c574457820be7f7bc29379d6d5e1cab51629810dfe2215bf4d37ed48aca829bd Jul 07 22:15:01.400722 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:01.400683 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" event={"ID":"9225687e-e709-4c9e-908d-95d707eb4d39","Type":"ContainerStarted","Data":"c574457820be7f7bc29379d6d5e1cab51629810dfe2215bf4d37ed48aca829bd"} Jul 07 22:15:04.419467 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:04.419423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" event={"ID":"9225687e-e709-4c9e-908d-95d707eb4d39","Type":"ContainerStarted","Data":"be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3"} Jul 07 22:15:04.434872 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:04.434817 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" podStartSLOduration=1.9364124139999999 podStartE2EDuration="4.434799943s" totalImagesPullingTime="2.498387529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:15:04.432992877 +0000 UTC m=+960.424566105" watchObservedRunningTime="2026-07-07 22:15:04.434799943 +0000 UTC m=+960.426373172" Jul 07 22:15:07.017743 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:15:07.017709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:16:00.012047 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:00.011991 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f"] Jul 07 22:16:00.012572 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:00.012251 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" podUID="9225687e-e709-4c9e-908d-95d707eb4d39" containerName="cleanup" containerID="cri-o://be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3" gracePeriod=30 Jul 07 22:16:30.194504 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.194470 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:16:30.244522 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.244474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kx6v\" (UniqueName: \"kubernetes.io/projected/9225687e-e709-4c9e-908d-95d707eb4d39-kube-api-access-6kx6v\") pod \"9225687e-e709-4c9e-908d-95d707eb4d39\" (UID: \"9225687e-e709-4c9e-908d-95d707eb4d39\") " Jul 07 22:16:30.246791 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.246750 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9225687e-e709-4c9e-908d-95d707eb4d39-kube-api-access-6kx6v" (OuterVolumeSpecName: "kube-api-access-6kx6v") pod "9225687e-e709-4c9e-908d-95d707eb4d39" (UID: "9225687e-e709-4c9e-908d-95d707eb4d39"). InnerVolumeSpecName "kube-api-access-6kx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:16:30.345503 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.345398 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kx6v\" (UniqueName: \"kubernetes.io/projected/9225687e-e709-4c9e-908d-95d707eb4d39-kube-api-access-6kx6v\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:16:30.853275 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.853240 2565 generic.go:356] "Generic (PLEG): container finished" podID="9225687e-e709-4c9e-908d-95d707eb4d39" containerID="be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3" exitCode=137 Jul 07 22:16:30.853530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.853287 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" Jul 07 22:16:30.853530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.853334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" event={"ID":"9225687e-e709-4c9e-908d-95d707eb4d39","Type":"ContainerDied","Data":"be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3"} Jul 07 22:16:30.853530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.853392 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f" event={"ID":"9225687e-e709-4c9e-908d-95d707eb4d39","Type":"ContainerDied","Data":"c574457820be7f7bc29379d6d5e1cab51629810dfe2215bf4d37ed48aca829bd"} Jul 07 22:16:30.853530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.853414 2565 scope.go:117] "RemoveContainer" containerID="be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3" Jul 07 22:16:30.862841 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.862823 2565 scope.go:117] "RemoveContainer" containerID="be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3" Jul 07 22:16:30.863211 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:16:30.863189 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3\": container with ID starting with be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3 not found: ID does not exist" containerID="be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3" Jul 07 22:16:30.863278 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.863223 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3"} err="failed to get container status \"be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3\": rpc error: code = NotFound desc = could not find container \"be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3\": container with ID starting with be6bcb9b799bef4253ab130307bd65cde57a0724b42a567893f55a28503756b3 not found: ID does not exist" Jul 07 22:16:30.868234 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.868182 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f"] Jul 07 22:16:30.869961 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:30.869940 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724375-55k2f"] Jul 07 22:16:32.576551 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:16:32.576520 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9225687e-e709-4c9e-908d-95d707eb4d39" path="/var/lib/kubelet/pods/9225687e-e709-4c9e-908d-95d707eb4d39/volumes" Jul 07 22:19:34.967453 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:19:34.967408 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:20:38.940754 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:20:38.940719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:21:51.991575 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:21:51.991535 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:23:29.273578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:23:29.273488 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:23:49.990929 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:23:49.990892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:03.041488 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:03.041446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:11.323396 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:11.323356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:16.485233 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:16.485197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:19.440152 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:19.440118 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:19.536530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:19.536471 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:19.885452 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:19.885410 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:25.646217 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:25.646117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:25.750079 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:25.750041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:25.849285 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:25.849245 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:25.904612 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:25.904527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:26.947503 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:26.947464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:27.015655 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.015505 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd"] Jul 07 22:24:27.016185 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.016158 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9225687e-e709-4c9e-908d-95d707eb4d39" containerName="cleanup" Jul 07 22:24:27.016185 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.016183 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9225687e-e709-4c9e-908d-95d707eb4d39" containerName="cleanup" Jul 07 22:24:27.016383 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.016292 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9225687e-e709-4c9e-908d-95d707eb4d39" containerName="cleanup" Jul 07 22:24:27.016487 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.016466 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:27.019444 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.019417 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.021251 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.021227 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a2c4a017-serving-cert\"" Jul 07 22:24:27.021393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.021294 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:24:27.028621 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.027972 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd"] Jul 07 22:24:27.045401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.045363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.045578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.045414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d67b5278-db25-4fe9-a30f-dc11e3c8e861-openshift-service-ca\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.045578 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.045551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gzs\" (UniqueName: \"kubernetes.io/projected/d67b5278-db25-4fe9-a30f-dc11e3c8e861-kube-api-access-f2gzs\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.074833 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.074798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:27.146448 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.146415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gzs\" (UniqueName: \"kubernetes.io/projected/d67b5278-db25-4fe9-a30f-dc11e3c8e861-kube-api-access-f2gzs\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.146668 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.146466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.146668 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.146496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d67b5278-db25-4fe9-a30f-dc11e3c8e861-openshift-service-ca\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.146668 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:24:27.146629 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-serving-cert: secret "maas-api-e2e-ait-a2c4a017-serving-cert" not found Jul 07 22:24:27.146832 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:24:27.146708 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls podName:d67b5278-db25-4fe9-a30f-dc11e3c8e861 nodeName:}" failed. No retries permitted until 2026-07-07 22:24:27.646686749 +0000 UTC m=+1523.638259958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls") pod "maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" (UID: "d67b5278-db25-4fe9-a30f-dc11e3c8e861") : secret "maas-api-e2e-ait-a2c4a017-serving-cert" not found Jul 07 22:24:27.147217 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.147194 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d67b5278-db25-4fe9-a30f-dc11e3c8e861-openshift-service-ca\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.159079 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.157336 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gzs\" (UniqueName: \"kubernetes.io/projected/d67b5278-db25-4fe9-a30f-dc11e3c8e861-kube-api-access-f2gzs\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.242009 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.241918 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:27.519304 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.519274 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:27.618410 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.618367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:27.651391 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.651355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.653869 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.653848 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls\") pod \"maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:27.933480 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:27.933379 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:28.080325 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:28.080288 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd"] Jul 07 22:24:28.082789 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:24:28.082753 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67b5278_db25_4fe9_a30f_dc11e3c8e861.slice/crio-ab541afdceab1908ef88957d7116241555af7f7bbba25c9e7a8764416aae3946 WatchSource:0}: Error finding container ab541afdceab1908ef88957d7116241555af7f7bbba25c9e7a8764416aae3946: Status 404 returned error can't find the container with id ab541afdceab1908ef88957d7116241555af7f7bbba25c9e7a8764416aae3946 Jul 07 22:24:28.084441 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:28.084419 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:24:28.208681 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:28.208592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" event={"ID":"d67b5278-db25-4fe9-a30f-dc11e3c8e861","Type":"ContainerStarted","Data":"ab541afdceab1908ef88957d7116241555af7f7bbba25c9e7a8764416aae3946"} Jul 07 22:24:29.259825 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:29.259789 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:30.221154 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:30.221117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" event={"ID":"d67b5278-db25-4fe9-a30f-dc11e3c8e861","Type":"ContainerStarted","Data":"79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c"} Jul 07 22:24:30.221348 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:30.221235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:30.237707 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:30.237648 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" podStartSLOduration=2.654148862 podStartE2EDuration="4.237631931s" totalImagesPullingTime="1.583483069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:24:30.235534866 +0000 UTC m=+1526.227108095" watchObservedRunningTime="2026-07-07 22:24:30.237631931 +0000 UTC m=+1526.229205158" Jul 07 22:24:32.195434 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.195398 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:32.345542 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.345505 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:32.449166 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.449081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:32.528845 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.528810 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd"] Jul 07 22:24:32.529046 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.529010 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" podUID="d67b5278-db25-4fe9-a30f-dc11e3c8e861" containerName="maas-api" containerID="cri-o://79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c" gracePeriod=30 Jul 07 22:24:32.551740 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.551696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:32.781928 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.781894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:32.800850 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.800803 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2gzs\" (UniqueName: \"kubernetes.io/projected/d67b5278-db25-4fe9-a30f-dc11e3c8e861-kube-api-access-f2gzs\") pod \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " Jul 07 22:24:32.801084 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.800892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls\") pod \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " Jul 07 22:24:32.801084 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.801042 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d67b5278-db25-4fe9-a30f-dc11e3c8e861-openshift-service-ca\") pod \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\" (UID: \"d67b5278-db25-4fe9-a30f-dc11e3c8e861\") " Jul 07 22:24:32.801437 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.801408 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67b5278-db25-4fe9-a30f-dc11e3c8e861-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d67b5278-db25-4fe9-a30f-dc11e3c8e861" (UID: "d67b5278-db25-4fe9-a30f-dc11e3c8e861"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:24:32.803381 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.803350 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d67b5278-db25-4fe9-a30f-dc11e3c8e861" (UID: "d67b5278-db25-4fe9-a30f-dc11e3c8e861"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:24:32.803650 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.803628 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67b5278-db25-4fe9-a30f-dc11e3c8e861-kube-api-access-f2gzs" (OuterVolumeSpecName: "kube-api-access-f2gzs") pod "d67b5278-db25-4fe9-a30f-dc11e3c8e861" (UID: "d67b5278-db25-4fe9-a30f-dc11e3c8e861"). InnerVolumeSpecName "kube-api-access-f2gzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:24:32.901904 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.901852 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d67b5278-db25-4fe9-a30f-dc11e3c8e861-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:24:32.901904 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.901895 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gzs\" (UniqueName: \"kubernetes.io/projected/d67b5278-db25-4fe9-a30f-dc11e3c8e861-kube-api-access-f2gzs\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:24:32.901904 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:32.901905 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d67b5278-db25-4fe9-a30f-dc11e3c8e861-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:24:33.237645 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.237606 2565 generic.go:356] "Generic (PLEG): container finished" podID="d67b5278-db25-4fe9-a30f-dc11e3c8e861" containerID="79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c" exitCode=0 Jul 07 22:24:33.238084 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.237657 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" Jul 07 22:24:33.238084 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.237686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" event={"ID":"d67b5278-db25-4fe9-a30f-dc11e3c8e861","Type":"ContainerDied","Data":"79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c"} Jul 07 22:24:33.238084 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.237738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd" event={"ID":"d67b5278-db25-4fe9-a30f-dc11e3c8e861","Type":"ContainerDied","Data":"ab541afdceab1908ef88957d7116241555af7f7bbba25c9e7a8764416aae3946"} Jul 07 22:24:33.238084 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.237771 2565 scope.go:117] "RemoveContainer" containerID="79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c" Jul 07 22:24:33.247545 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.247523 2565 scope.go:117] "RemoveContainer" containerID="79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c" Jul 07 22:24:33.247887 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:24:33.247867 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c\": container with ID starting with 79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c not found: ID does not exist" containerID="79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c" Jul 07 22:24:33.247939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.247897 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c"} err="failed to get container status \"79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c\": rpc error: code = NotFound desc = could not find container \"79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c\": container with ID starting with 79b388b954067cad122e81689f188b6770ae772f9685552ba990cbf571cf369c not found: ID does not exist" Jul 07 22:24:33.257643 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.257605 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd"] Jul 07 22:24:33.261712 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.261684 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2c4a017-89f556f6-4nqmd"] Jul 07 22:24:33.870122 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:33.870014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:34.582857 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:34.582820 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67b5278-db25-4fe9-a30f-dc11e3c8e861" path="/var/lib/kubelet/pods/d67b5278-db25-4fe9-a30f-dc11e3c8e861/volumes" Jul 07 22:24:38.160089 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:38.160052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:38.261859 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:38.261824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:38.368104 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:38.368071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:38.428370 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:38.428287 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:39.473127 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.473089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:39.536075 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.536039 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6"] Jul 07 22:24:39.536484 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.536471 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67b5278-db25-4fe9-a30f-dc11e3c8e861" containerName="maas-api" Jul 07 22:24:39.536530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.536486 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67b5278-db25-4fe9-a30f-dc11e3c8e861" containerName="maas-api" Jul 07 22:24:39.536583 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.536574 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67b5278-db25-4fe9-a30f-dc11e3c8e861" containerName="maas-api" Jul 07 22:24:39.541398 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.541372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.543394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.543358 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1d13ad38-serving-cert\"" Jul 07 22:24:39.543531 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.543440 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:24:39.549476 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.549447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6"] Jul 07 22:24:39.551373 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.551350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:39.616803 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.616768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:39.662293 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.662239 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-maas-api-tls\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.662478 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.662327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-openshift-service-ca\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.662478 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.662349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwczw\" (UniqueName: \"kubernetes.io/projected/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-kube-api-access-fwczw\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.684298 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.684247 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:39.763291 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.763254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-openshift-service-ca\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.763458 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.763298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwczw\" (UniqueName: \"kubernetes.io/projected/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-kube-api-access-fwczw\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.763458 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.763381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-maas-api-tls\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.764056 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.764003 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-openshift-service-ca\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.766080 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.766055 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-maas-api-tls\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.771311 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.771284 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwczw\" (UniqueName: \"kubernetes.io/projected/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-kube-api-access-fwczw\") pod \"maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:39.855249 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:39.855211 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:40.000765 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:40.000720 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6"] Jul 07 22:24:40.004236 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:24:40.004204 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a5e9c6_8949_4b3c_b761_f8fe7f1ad4eb.slice/crio-c9ce2bc6a4c5a483f2fcdbdfc945d61561cafdbc0e19706dbe1177d6ba930587 WatchSource:0}: Error finding container c9ce2bc6a4c5a483f2fcdbdfc945d61561cafdbc0e19706dbe1177d6ba930587: Status 404 returned error can't find the container with id c9ce2bc6a4c5a483f2fcdbdfc945d61561cafdbc0e19706dbe1177d6ba930587 Jul 07 22:24:40.020930 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:40.020819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:40.128969 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:40.128923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:40.276124 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:40.276011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" event={"ID":"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb","Type":"ContainerStarted","Data":"c9ce2bc6a4c5a483f2fcdbdfc945d61561cafdbc0e19706dbe1177d6ba930587"} Jul 07 22:24:41.283074 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:41.283010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" event={"ID":"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb","Type":"ContainerStarted","Data":"f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee"} Jul 07 22:24:41.283476 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:41.283118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:41.301001 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:41.300939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" podStartSLOduration=1.88519211 podStartE2EDuration="2.300922525s" totalImagesPullingTime="415.730415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:24:41.299294834 +0000 UTC m=+1537.290868064" watchObservedRunningTime="2026-07-07 22:24:41.300922525 +0000 UTC m=+1537.292495753" Jul 07 22:24:41.579369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:41.579335 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:44.661765 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:44.661727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:44.956195 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:44.956108 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6"] Jul 07 22:24:44.956383 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:44.956329 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" podUID="a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" containerName="maas-api" containerID="cri-o://f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee" gracePeriod=30 Jul 07 22:24:44.981258 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:44.981192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:45.210291 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.210220 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:45.304014 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.303982 2565 generic.go:356] "Generic (PLEG): container finished" podID="a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" containerID="f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee" exitCode=0 Jul 07 22:24:45.304183 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.304043 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" Jul 07 22:24:45.304183 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.304067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" event={"ID":"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb","Type":"ContainerDied","Data":"f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee"} Jul 07 22:24:45.304183 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.304126 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6" event={"ID":"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb","Type":"ContainerDied","Data":"c9ce2bc6a4c5a483f2fcdbdfc945d61561cafdbc0e19706dbe1177d6ba930587"} Jul 07 22:24:45.304183 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.304156 2565 scope.go:117] "RemoveContainer" containerID="f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee" Jul 07 22:24:45.313671 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.313651 2565 scope.go:117] "RemoveContainer" containerID="f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee" Jul 07 22:24:45.314058 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:24:45.314011 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee\": container with ID starting with f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee not found: ID does not exist" containerID="f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee" Jul 07 22:24:45.314158 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.314069 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee"} err="failed to get container status \"f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee\": rpc error: code = NotFound desc = could not find container \"f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee\": container with ID starting with f0cb2622a6ff05b719614c80634dda78e3ae95676a1f333024974876c0b1aaee not found: ID does not exist" Jul 07 22:24:45.320443 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.320418 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwczw\" (UniqueName: \"kubernetes.io/projected/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-kube-api-access-fwczw\") pod \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " Jul 07 22:24:45.320541 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.320465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-maas-api-tls\") pod \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " Jul 07 22:24:45.320541 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.320516 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-openshift-service-ca\") pod \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\" (UID: \"a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb\") " Jul 07 22:24:45.320923 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.320901 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" (UID: "a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:24:45.322766 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.322729 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" (UID: "a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:24:45.322766 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.322755 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-kube-api-access-fwczw" (OuterVolumeSpecName: "kube-api-access-fwczw") pod "a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" (UID: "a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb"). InnerVolumeSpecName "kube-api-access-fwczw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:24:45.421552 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.421506 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:24:45.421552 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.421545 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwczw\" (UniqueName: \"kubernetes.io/projected/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-kube-api-access-fwczw\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:24:45.421552 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.421555 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:24:45.625538 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.625500 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6"] Jul 07 22:24:45.628559 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:45.628527 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d13ad38-5bddcff5db-8czk6"] Jul 07 22:24:46.577619 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:46.577574 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" path="/var/lib/kubelet/pods/a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb/volumes" Jul 07 22:24:50.794716 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:50.794678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:50.889675 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:50.889636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:53.330334 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:53.330301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:56.495359 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:56.495322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:56.559737 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:56.559704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:56.698319 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:56.698263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:56.756640 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:56.756602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:57.811714 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:57.811682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:57.907546 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:57.907506 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:58.009334 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.009300 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd"] Jul 07 22:24:58.009733 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.009721 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" containerName="maas-api" Jul 07 22:24:58.009778 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.009735 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" containerName="maas-api" Jul 07 22:24:58.009813 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.009789 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a5e9c6-8949-4b3c-b761-f8fe7f1ad4eb" containerName="maas-api" Jul 07 22:24:58.014558 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.014539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.016579 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.016550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e6aec1d1-serving-cert\"" Jul 07 22:24:58.016579 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.016575 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:24:58.021653 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.021610 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd"] Jul 07 22:24:58.115201 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.115105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:58.153523 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.153486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.153675 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.153548 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-openshift-service-ca\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.153675 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.153628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kfz\" (UniqueName: \"kubernetes.io/projected/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-kube-api-access-h5kfz\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.254373 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.254330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.254559 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.254402 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-openshift-service-ca\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.254559 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.254426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kfz\" (UniqueName: \"kubernetes.io/projected/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-kube-api-access-h5kfz\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.254559 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:24:58.254492 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-serving-cert: secret "maas-api-e2e-derive-e6aec1d1-serving-cert" not found Jul 07 22:24:58.254683 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:24:58.254559 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls podName:a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4 nodeName:}" failed. No retries permitted until 2026-07-07 22:24:58.754539013 +0000 UTC m=+1554.746112220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls") pod "maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" (UID: "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4") : secret "maas-api-e2e-derive-e6aec1d1-serving-cert" not found Jul 07 22:24:58.255088 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.255067 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-openshift-service-ca\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.265955 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.265919 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kfz\" (UniqueName: \"kubernetes.io/projected/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-kube-api-access-h5kfz\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.522945 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.522907 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:58.623400 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.623365 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:58.725625 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.725593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:24:58.759996 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.759947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.762730 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.762701 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls\") pod \"maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:58.927891 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:58.927782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:24:59.077268 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:59.077144 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd"] Jul 07 22:24:59.080409 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:24:59.080379 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cabe3f_ee8d_4eda_8abe_28b3702cf4a4.slice/crio-eb5261e7241dd42a0de2ee4db6dc449f6b96af725ee73619c054879be3955d20 WatchSource:0}: Error finding container eb5261e7241dd42a0de2ee4db6dc449f6b96af725ee73619c054879be3955d20: Status 404 returned error can't find the container with id eb5261e7241dd42a0de2ee4db6dc449f6b96af725ee73619c054879be3955d20 Jul 07 22:24:59.381617 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:24:59.381570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" event={"ID":"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4","Type":"ContainerStarted","Data":"eb5261e7241dd42a0de2ee4db6dc449f6b96af725ee73619c054879be3955d20"} Jul 07 22:25:00.238613 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:00.238572 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:00.387910 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:00.387869 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" event={"ID":"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4","Type":"ContainerStarted","Data":"adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307"} Jul 07 22:25:00.388172 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:00.387984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:25:00.404891 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:00.404833 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" podStartSLOduration=3.021395422 podStartE2EDuration="3.404813883s" totalImagesPullingTime="383.418461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:25:00.402978509 +0000 UTC m=+1556.394551740" watchObservedRunningTime="2026-07-07 22:25:00.404813883 +0000 UTC m=+1556.396387111" Jul 07 22:25:02.261716 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.261685 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:02.459223 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.459184 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:02.561976 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.561876 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:02.577954 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.577923 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd"] Jul 07 22:25:02.578202 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.578177 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" podUID="a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" containerName="maas-api" containerID="cri-o://adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307" gracePeriod=30 Jul 07 22:25:02.615306 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.615274 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:02.834329 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:02.834302 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:25:03.002384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.002336 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-openshift-service-ca\") pod \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " Jul 07 22:25:03.002384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.002387 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5kfz\" (UniqueName: \"kubernetes.io/projected/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-kube-api-access-h5kfz\") pod \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " Jul 07 22:25:03.002595 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.002414 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls\") pod \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\" (UID: \"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4\") " Jul 07 22:25:03.002887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.002866 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" (UID: "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:25:03.004904 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.004875 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-kube-api-access-h5kfz" (OuterVolumeSpecName: "kube-api-access-h5kfz") pod "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" (UID: "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4"). InnerVolumeSpecName "kube-api-access-h5kfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:25:03.004987 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.004881 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" (UID: "a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:25:03.103391 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.103293 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:25:03.103391 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.103325 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kfz\" (UniqueName: \"kubernetes.io/projected/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-kube-api-access-h5kfz\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:25:03.103391 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.103336 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:25:03.405294 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.405198 2565 generic.go:356] "Generic (PLEG): container finished" podID="a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" containerID="adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307" exitCode=0 Jul 07 22:25:03.405294 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.405240 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" Jul 07 22:25:03.405294 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.405275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" event={"ID":"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4","Type":"ContainerDied","Data":"adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307"} Jul 07 22:25:03.405820 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.405328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd" event={"ID":"a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4","Type":"ContainerDied","Data":"eb5261e7241dd42a0de2ee4db6dc449f6b96af725ee73619c054879be3955d20"} Jul 07 22:25:03.405820 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.405349 2565 scope.go:117] "RemoveContainer" containerID="adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307" Jul 07 22:25:03.414428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.414407 2565 scope.go:117] "RemoveContainer" containerID="adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307" Jul 07 22:25:03.414724 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:25:03.414704 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307\": container with ID starting with adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307 not found: ID does not exist" containerID="adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307" Jul 07 22:25:03.414783 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.414733 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307"} err="failed to get container status \"adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307\": rpc error: code = NotFound desc = could not find container \"adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307\": container with ID starting with adafcdb87b9357889f074edda26ba59304ccd26432e387d9335e520e3609d307 not found: ID does not exist" Jul 07 22:25:03.425565 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.425532 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd"] Jul 07 22:25:03.429109 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:03.429084 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6aec1d1-5456f4fcb8-zzvnd"] Jul 07 22:25:04.036853 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:04.036804 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:04.576586 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:04.576552 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" path="/var/lib/kubelet/pods/a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4/volumes" Jul 07 22:25:08.477320 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:08.477275 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:08.548270 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:08.548239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:08.948201 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:08.948170 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:09.047658 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:09.047612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:09.147376 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:09.147328 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:10.552630 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:10.552592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:16.279158 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:16.279120 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:16.335974 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:16.335935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:16.524202 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:16.524170 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:16.610615 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:16.610528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:17.008441 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:17.008399 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:42.994461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:42.994427 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:48.943544 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:48.943504 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:49.005452 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:49.005421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:49.089655 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:49.089591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:25:49.191466 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:25:49.191421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:05.060286 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:05.060189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:10.763746 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:10.763705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:10.828212 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:10.828169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:10.902362 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:10.902334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:11.002354 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:11.002317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:21.493623 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:21.493554 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:22.294152 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:22.294117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:27.955839 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:27.955801 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:28.021878 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:28.021842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:28.104608 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:28.104532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:28.217653 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:28.217571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:28.316303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:28.316271 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:29.049956 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:29.049906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:29.149864 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:29.149831 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:29.250011 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:29.249974 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:30.930429 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:30.930392 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:36.598929 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:36.598893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:36.699374 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:36.699330 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:37.300141 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:37.300105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:43.141753 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:43.141715 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:43.217237 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:43.217206 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:43.390540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:43.390503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:43.828852 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:43.828807 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:44.067433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:44.067399 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:46.957958 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:46.957920 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:47.057330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.057294 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:47.126795 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.126756 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2"] Jul 07 22:26:47.127399 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.127370 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" containerName="maas-api" Jul 07 22:26:47.127399 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.127398 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" containerName="maas-api" Jul 07 22:26:47.127583 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.127512 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cabe3f-ee8d-4eda-8abe-28b3702cf4a4" containerName="maas-api" Jul 07 22:26:47.130693 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.130661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.132689 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.132663 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d19c1ee-serving-cert\"" Jul 07 22:26:47.132849 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.132758 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:26:47.138644 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.138607 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2"] Jul 07 22:26:47.158108 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.158070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:47.189703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.189658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.189703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.189709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c75ffc-b490-4b7e-a40c-240af957ee05-openshift-service-ca\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.189958 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.189740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wv2\" (UniqueName: \"kubernetes.io/projected/e6c75ffc-b490-4b7e-a40c-240af957ee05-kube-api-access-s6wv2\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.290963 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.290925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.291211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.290976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c75ffc-b490-4b7e-a40c-240af957ee05-openshift-service-ca\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.291211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.291004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wv2\" (UniqueName: \"kubernetes.io/projected/e6c75ffc-b490-4b7e-a40c-240af957ee05-kube-api-access-s6wv2\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.291211 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:26:47.291132 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-serving-cert: secret "maas-api-e2e-mt-3d19c1ee-serving-cert" not found Jul 07 22:26:47.291364 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:26:47.291232 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls podName:e6c75ffc-b490-4b7e-a40c-240af957ee05 nodeName:}" failed. No retries permitted until 2026-07-07 22:26:47.791211168 +0000 UTC m=+1663.782784389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls") pod "maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" (UID: "e6c75ffc-b490-4b7e-a40c-240af957ee05") : secret "maas-api-e2e-mt-3d19c1ee-serving-cert" not found Jul 07 22:26:47.291736 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.291715 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c75ffc-b490-4b7e-a40c-240af957ee05-openshift-service-ca\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.299725 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.299692 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wv2\" (UniqueName: \"kubernetes.io/projected/e6c75ffc-b490-4b7e-a40c-240af957ee05-kube-api-access-s6wv2\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.611297 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.611201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:47.711433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.711390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:47.796677 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.796635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.799283 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.799261 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls\") pod \"maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:47.814574 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:47.814540 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:48.043993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:48.043949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:48.194160 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:48.194110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2"] Jul 07 22:26:48.196697 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:26:48.196665 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c75ffc_b490_4b7e_a40c_240af957ee05.slice/crio-e0ed9ce623f5ff9827db4e9005310e76990fde9dc86bed3a28eaff2219aa0e38 WatchSource:0}: Error finding container e0ed9ce623f5ff9827db4e9005310e76990fde9dc86bed3a28eaff2219aa0e38: Status 404 returned error can't find the container with id e0ed9ce623f5ff9827db4e9005310e76990fde9dc86bed3a28eaff2219aa0e38 Jul 07 22:26:48.940058 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:48.939992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" event={"ID":"e6c75ffc-b490-4b7e-a40c-240af957ee05","Type":"ContainerStarted","Data":"a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169"} Jul 07 22:26:48.940058 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:48.940061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" event={"ID":"e6c75ffc-b490-4b7e-a40c-240af957ee05","Type":"ContainerStarted","Data":"e0ed9ce623f5ff9827db4e9005310e76990fde9dc86bed3a28eaff2219aa0e38"} Jul 07 22:26:48.940323 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:48.940094 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:48.956402 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:48.956290 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" podStartSLOduration=1.462436708 podStartE2EDuration="1.956266206s" totalImagesPullingTime="493.829498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:26:48.955269857 +0000 UTC m=+1664.946843086" watchObservedRunningTime="2026-07-07 22:26:48.956266206 +0000 UTC m=+1664.947839433" Jul 07 22:26:51.498715 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:51.498670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:52.214051 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:52.213991 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:52.312496 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:52.312464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:52.628559 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:52.628529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:54.952896 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:54.952869 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:58.505979 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:58.505941 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:58.565335 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:58.565300 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:58.911271 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:58.911174 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:59.230538 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.230436 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2"] Jul 07 22:26:59.230753 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.230694 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" podUID="e6c75ffc-b490-4b7e-a40c-240af957ee05" containerName="maas-api" containerID="cri-o://a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169" gracePeriod=30 Jul 07 22:26:59.253852 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.253817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:59.353732 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.353695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:26:59.483541 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.483461 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:59.606821 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.606779 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c75ffc-b490-4b7e-a40c-240af957ee05-openshift-service-ca\") pod \"e6c75ffc-b490-4b7e-a40c-240af957ee05\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " Jul 07 22:26:59.606821 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.606829 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls\") pod \"e6c75ffc-b490-4b7e-a40c-240af957ee05\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " Jul 07 22:26:59.607376 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.606879 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6wv2\" (UniqueName: \"kubernetes.io/projected/e6c75ffc-b490-4b7e-a40c-240af957ee05-kube-api-access-s6wv2\") pod \"e6c75ffc-b490-4b7e-a40c-240af957ee05\" (UID: \"e6c75ffc-b490-4b7e-a40c-240af957ee05\") " Jul 07 22:26:59.607376 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.607235 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c75ffc-b490-4b7e-a40c-240af957ee05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6c75ffc-b490-4b7e-a40c-240af957ee05" (UID: "e6c75ffc-b490-4b7e-a40c-240af957ee05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:26:59.609245 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.609218 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c75ffc-b490-4b7e-a40c-240af957ee05-kube-api-access-s6wv2" (OuterVolumeSpecName: "kube-api-access-s6wv2") pod "e6c75ffc-b490-4b7e-a40c-240af957ee05" (UID: "e6c75ffc-b490-4b7e-a40c-240af957ee05"). InnerVolumeSpecName "kube-api-access-s6wv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:26:59.609312 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.609219 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6c75ffc-b490-4b7e-a40c-240af957ee05" (UID: "e6c75ffc-b490-4b7e-a40c-240af957ee05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:26:59.708352 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.708308 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c75ffc-b490-4b7e-a40c-240af957ee05-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:26:59.708352 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.708344 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c75ffc-b490-4b7e-a40c-240af957ee05-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:26:59.708352 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.708357 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wv2\" (UniqueName: \"kubernetes.io/projected/e6c75ffc-b490-4b7e-a40c-240af957ee05-kube-api-access-s6wv2\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:26:59.995388 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.995353 2565 generic.go:356] "Generic (PLEG): container finished" podID="e6c75ffc-b490-4b7e-a40c-240af957ee05" containerID="a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169" exitCode=0 Jul 07 22:26:59.995540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.995395 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" Jul 07 22:26:59.995540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.995450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" event={"ID":"e6c75ffc-b490-4b7e-a40c-240af957ee05","Type":"ContainerDied","Data":"a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169"} Jul 07 22:26:59.995540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.995502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2" event={"ID":"e6c75ffc-b490-4b7e-a40c-240af957ee05","Type":"ContainerDied","Data":"e0ed9ce623f5ff9827db4e9005310e76990fde9dc86bed3a28eaff2219aa0e38"} Jul 07 22:26:59.995540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:26:59.995524 2565 scope.go:117] "RemoveContainer" containerID="a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169" Jul 07 22:27:00.004792 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:00.004767 2565 scope.go:117] "RemoveContainer" containerID="a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169" Jul 07 22:27:00.005121 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:27:00.005102 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169\": container with ID starting with a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169 not found: ID does not exist" containerID="a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169" Jul 07 22:27:00.005173 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:00.005129 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169"} err="failed to get container status \"a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169\": rpc error: code = NotFound desc = could not find container \"a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169\": container with ID starting with a1b4692daa48d36bc38994386c4e0af42f96d835d7a399d74f03a6925c02a169 not found: ID does not exist" Jul 07 22:27:00.015577 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:00.015498 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2"] Jul 07 22:27:00.017443 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:00.017419 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d19c1ee-668d7b6d49-t6ns2"] Jul 07 22:27:00.577170 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:00.577134 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c75ffc-b490-4b7e-a40c-240af957ee05" path="/var/lib/kubelet/pods/e6c75ffc-b490-4b7e-a40c-240af957ee05/volumes" Jul 07 22:27:05.035775 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:05.035737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:05.136107 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:05.136070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:05.335146 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:05.335018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:05.658487 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:05.658388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:05.757907 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:05.757871 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:05.873428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:05.873394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:11.679473 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:11.679437 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:11.736188 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:11.736146 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:11.818568 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:11.818457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:11.916298 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:11.916265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:12.019417 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:12.019378 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:17.760390 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:17.760361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:17.860598 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:17.860555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:23.196811 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:23.196771 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:23.298096 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:23.298061 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:24.548367 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:24.548284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:24.916332 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:24.916237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:25.054255 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:25.054215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.127203 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.127154 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.200169 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.200133 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.205320 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.205290 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td"] Jul 07 22:27:28.205784 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.205759 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c75ffc-b490-4b7e-a40c-240af957ee05" containerName="maas-api" Jul 07 22:27:28.205784 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.205784 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c75ffc-b490-4b7e-a40c-240af957ee05" containerName="maas-api" Jul 07 22:27:28.206003 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.205895 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c75ffc-b490-4b7e-a40c-240af957ee05" containerName="maas-api" Jul 07 22:27:28.209325 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.209305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.211241 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.211220 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5ebf133b-serving-cert\"" Jul 07 22:27:28.211352 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.211269 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:27:28.217958 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.217887 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td"] Jul 07 22:27:28.264157 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.264127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.285949 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.285914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.286110 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.285953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xcs\" (UniqueName: \"kubernetes.io/projected/5b44c207-2889-4d2d-b32e-88a6650fa37a-kube-api-access-46xcs\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.286110 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.285979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b44c207-2889-4d2d-b32e-88a6650fa37a-openshift-service-ca\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.386914 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.386812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.386914 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.386858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xcs\" (UniqueName: \"kubernetes.io/projected/5b44c207-2889-4d2d-b32e-88a6650fa37a-kube-api-access-46xcs\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.386914 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.386890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b44c207-2889-4d2d-b32e-88a6650fa37a-openshift-service-ca\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.387187 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:27:28.386978 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-serving-cert: secret "maas-api-e2e-mt-5ebf133b-serving-cert" not found Jul 07 22:27:28.387187 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:27:28.387100 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls podName:5b44c207-2889-4d2d-b32e-88a6650fa37a nodeName:}" failed. No retries permitted until 2026-07-07 22:27:28.887076624 +0000 UTC m=+1704.878649842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls") pod "maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" (UID: "5b44c207-2889-4d2d-b32e-88a6650fa37a") : secret "maas-api-e2e-mt-5ebf133b-serving-cert" not found Jul 07 22:27:28.387624 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.387604 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b44c207-2889-4d2d-b32e-88a6650fa37a-openshift-service-ca\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.395264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.395234 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xcs\" (UniqueName: \"kubernetes.io/projected/5b44c207-2889-4d2d-b32e-88a6650fa37a-kube-api-access-46xcs\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.660497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.660404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.771018 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.770980 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.871478 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.871445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:28.891691 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.891657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:28.894265 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:28.894242 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls\") pod \"maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:29.121682 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:29.121647 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:29.268774 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:29.268744 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td"] Jul 07 22:27:29.271189 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:27:29.271158 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b44c207_2889_4d2d_b32e_88a6650fa37a.slice/crio-b068a6461e54e89243c79eb7e46d7493c3dd342eb4aa31da49c1f512fe961ec5 WatchSource:0}: Error finding container b068a6461e54e89243c79eb7e46d7493c3dd342eb4aa31da49c1f512fe961ec5: Status 404 returned error can't find the container with id b068a6461e54e89243c79eb7e46d7493c3dd342eb4aa31da49c1f512fe961ec5 Jul 07 22:27:30.146604 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:30.146552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" event={"ID":"5b44c207-2889-4d2d-b32e-88a6650fa37a","Type":"ContainerStarted","Data":"5e5c580febd1e44ab8de56c4731043cc5e742b362eb213ff3e3adb05a90b88ae"} Jul 07 22:27:30.146809 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:30.146643 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" event={"ID":"5b44c207-2889-4d2d-b32e-88a6650fa37a","Type":"ContainerStarted","Data":"b068a6461e54e89243c79eb7e46d7493c3dd342eb4aa31da49c1f512fe961ec5"} Jul 07 22:27:30.146809 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:30.146667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:30.165241 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:30.165184 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" podStartSLOduration=1.72162627 podStartE2EDuration="2.165166886s" totalImagesPullingTime="443.540616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:27:30.16300742 +0000 UTC m=+1706.154580650" watchObservedRunningTime="2026-07-07 22:27:30.165166886 +0000 UTC m=+1706.156740114" Jul 07 22:27:32.639348 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:32.639313 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:35.551203 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.551164 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l"] Jul 07 22:27:35.554700 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.554670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.556702 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.556673 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5ebf133b\"/\"default-dockercfg-kjnqm\"" Jul 07 22:27:35.557366 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.557347 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5ebf133b\"/\"openshift-service-ca.crt\"" Jul 07 22:27:35.557481 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.557348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5ebf133b\"/\"e2e-lifecycle-model-5ebf133b-kserve-self-signed-certs\"" Jul 07 22:27:35.557481 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.557358 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5ebf133b\"/\"kube-root-ca.crt\"" Jul 07 22:27:35.562206 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.562165 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l"] Jul 07 22:27:35.657476 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.657435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-home\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.657476 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.657472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-model-cache\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.657753 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.657579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tmp-dir\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.657753 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.657688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4hc\" (UniqueName: \"kubernetes.io/projected/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-kube-api-access-tt4hc\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.657867 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.657843 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-dshm\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.657975 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.657954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tls-certs\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.758759 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.758723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-dshm\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.758959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.758771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tls-certs\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.758959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.758801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-home\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.758959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.758817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-model-cache\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.758959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.758864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tmp-dir\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.758959 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.758911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4hc\" (UniqueName: \"kubernetes.io/projected/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-kube-api-access-tt4hc\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.759361 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.759325 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-home\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.759428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.759397 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-model-cache\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.759428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.759409 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tmp-dir\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.761333 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.761305 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-dshm\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.761521 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.761502 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tls-certs\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.767568 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.767539 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4hc\" (UniqueName: \"kubernetes.io/projected/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-kube-api-access-tt4hc\") pod \"e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:35.852906 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.852823 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:35.866512 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:35.866474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:36.006516 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:36.006311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l"] Jul 07 22:27:36.009631 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:27:36.009592 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc39b4a5_08af_4ed9_8b51_9527d55ee6da.slice/crio-0a5f7f906ee31491d7567af180f388b3eba130a2165c3fcce7d6beec1364dc3f WatchSource:0}: Error finding container 0a5f7f906ee31491d7567af180f388b3eba130a2165c3fcce7d6beec1364dc3f: Status 404 returned error can't find the container with id 0a5f7f906ee31491d7567af180f388b3eba130a2165c3fcce7d6beec1364dc3f Jul 07 22:27:36.157708 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:36.157632 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:36.188870 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:36.188805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" event={"ID":"dc39b4a5-08af-4ed9-8b51-9527d55ee6da","Type":"ContainerStarted","Data":"cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777"} Jul 07 22:27:36.189256 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:36.189224 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" event={"ID":"dc39b4a5-08af-4ed9-8b51-9527d55ee6da","Type":"ContainerStarted","Data":"0a5f7f906ee31491d7567af180f388b3eba130a2165c3fcce7d6beec1364dc3f"} Jul 07 22:27:45.867153 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:45.867108 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:45.867153 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:45.867165 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:45.880205 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:45.880167 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:45.895482 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:45.895431 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" podStartSLOduration=10.895413326 podStartE2EDuration="10.895413326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:27:36.204534308 +0000 UTC m=+1712.196107536" watchObservedRunningTime="2026-07-07 22:27:45.895413326 +0000 UTC m=+1721.886986552" Jul 07 22:27:46.251681 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:46.251648 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:27:54.809291 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:54.809256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:55.113868 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.113769 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td"] Jul 07 22:27:55.114066 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.114044 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" podUID="5b44c207-2889-4d2d-b32e-88a6650fa37a" containerName="maas-api" containerID="cri-o://5e5c580febd1e44ab8de56c4731043cc5e742b362eb213ff3e3adb05a90b88ae" gracePeriod=30 Jul 07 22:27:55.138378 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.138339 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:27:55.289259 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.289227 2565 generic.go:356] "Generic (PLEG): container finished" podID="5b44c207-2889-4d2d-b32e-88a6650fa37a" containerID="5e5c580febd1e44ab8de56c4731043cc5e742b362eb213ff3e3adb05a90b88ae" exitCode=0 Jul 07 22:27:55.289401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.289292 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" event={"ID":"5b44c207-2889-4d2d-b32e-88a6650fa37a","Type":"ContainerDied","Data":"5e5c580febd1e44ab8de56c4731043cc5e742b362eb213ff3e3adb05a90b88ae"} Jul 07 22:27:55.368405 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.368336 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:55.452520 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.452479 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46xcs\" (UniqueName: \"kubernetes.io/projected/5b44c207-2889-4d2d-b32e-88a6650fa37a-kube-api-access-46xcs\") pod \"5b44c207-2889-4d2d-b32e-88a6650fa37a\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " Jul 07 22:27:55.452520 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.452527 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b44c207-2889-4d2d-b32e-88a6650fa37a-openshift-service-ca\") pod \"5b44c207-2889-4d2d-b32e-88a6650fa37a\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " Jul 07 22:27:55.452742 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.452624 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls\") pod \"5b44c207-2889-4d2d-b32e-88a6650fa37a\" (UID: \"5b44c207-2889-4d2d-b32e-88a6650fa37a\") " Jul 07 22:27:55.452982 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.452952 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b44c207-2889-4d2d-b32e-88a6650fa37a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b44c207-2889-4d2d-b32e-88a6650fa37a" (UID: "5b44c207-2889-4d2d-b32e-88a6650fa37a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:27:55.454912 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.454885 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b44c207-2889-4d2d-b32e-88a6650fa37a-kube-api-access-46xcs" (OuterVolumeSpecName: "kube-api-access-46xcs") pod "5b44c207-2889-4d2d-b32e-88a6650fa37a" (UID: "5b44c207-2889-4d2d-b32e-88a6650fa37a"). InnerVolumeSpecName "kube-api-access-46xcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:27:55.455074 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.455005 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b44c207-2889-4d2d-b32e-88a6650fa37a" (UID: "5b44c207-2889-4d2d-b32e-88a6650fa37a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:27:55.553952 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.553907 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b44c207-2889-4d2d-b32e-88a6650fa37a-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:27:55.553952 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.553942 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b44c207-2889-4d2d-b32e-88a6650fa37a-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:27:55.553952 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:55.553957 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xcs\" (UniqueName: \"kubernetes.io/projected/5b44c207-2889-4d2d-b32e-88a6650fa37a-kube-api-access-46xcs\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:27:56.295506 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:56.295473 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" Jul 07 22:27:56.295506 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:56.295485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td" event={"ID":"5b44c207-2889-4d2d-b32e-88a6650fa37a","Type":"ContainerDied","Data":"b068a6461e54e89243c79eb7e46d7493c3dd342eb4aa31da49c1f512fe961ec5"} Jul 07 22:27:56.296008 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:56.295589 2565 scope.go:117] "RemoveContainer" containerID="5e5c580febd1e44ab8de56c4731043cc5e742b362eb213ff3e3adb05a90b88ae" Jul 07 22:27:56.316948 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:56.316910 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td"] Jul 07 22:27:56.320165 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:56.320136 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ebf133b-cbd477fd5-2r8td"] Jul 07 22:27:56.577301 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:27:56.577214 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b44c207-2889-4d2d-b32e-88a6650fa37a" path="/var/lib/kubelet/pods/5b44c207-2889-4d2d-b32e-88a6650fa37a/volumes" Jul 07 22:28:00.907458 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:00.907413 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:05.890659 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:05.890615 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l"] Jul 07 22:28:05.933622 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:05.933587 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:20.912330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:20.912284 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" podUID="dc39b4a5-08af-4ed9-8b51-9527d55ee6da" containerName="main" containerID="cri-o://cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777" gracePeriod=45 Jul 07 22:28:21.175796 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.175715 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:28:21.310843 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.310799 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt4hc\" (UniqueName: \"kubernetes.io/projected/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-kube-api-access-tt4hc\") pod \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " Jul 07 22:28:21.311129 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.310856 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tmp-dir\") pod \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " Jul 07 22:28:21.311129 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.310872 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-dshm\") pod \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " Jul 07 22:28:21.311129 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.310906 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-model-cache\") pod \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " Jul 07 22:28:21.311129 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.310972 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-home\") pod \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " Jul 07 22:28:21.311129 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311099 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tls-certs\") pod \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\" (UID: \"dc39b4a5-08af-4ed9-8b51-9527d55ee6da\") " Jul 07 22:28:21.311401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311192 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc39b4a5-08af-4ed9-8b51-9527d55ee6da" (UID: "dc39b4a5-08af-4ed9-8b51-9527d55ee6da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:28:21.311401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311274 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc39b4a5-08af-4ed9-8b51-9527d55ee6da" (UID: "dc39b4a5-08af-4ed9-8b51-9527d55ee6da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:28:21.311401 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311290 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-home" (OuterVolumeSpecName: "home") pod "dc39b4a5-08af-4ed9-8b51-9527d55ee6da" (UID: "dc39b4a5-08af-4ed9-8b51-9527d55ee6da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:28:21.311548 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311475 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tmp-dir\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:28:21.311548 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311491 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-model-cache\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:28:21.311548 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.311499 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-home\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:28:21.313439 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.313417 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-dshm" (OuterVolumeSpecName: "dshm") pod "dc39b4a5-08af-4ed9-8b51-9527d55ee6da" (UID: "dc39b4a5-08af-4ed9-8b51-9527d55ee6da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:28:21.313527 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.313454 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-kube-api-access-tt4hc" (OuterVolumeSpecName: "kube-api-access-tt4hc") pod "dc39b4a5-08af-4ed9-8b51-9527d55ee6da" (UID: "dc39b4a5-08af-4ed9-8b51-9527d55ee6da"). InnerVolumeSpecName "kube-api-access-tt4hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:28:21.313527 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.313453 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc39b4a5-08af-4ed9-8b51-9527d55ee6da" (UID: "dc39b4a5-08af-4ed9-8b51-9527d55ee6da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:28:21.412263 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.412216 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-tls-certs\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:28:21.412263 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.412254 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4hc\" (UniqueName: \"kubernetes.io/projected/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-kube-api-access-tt4hc\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:28:21.412263 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.412264 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39b4a5-08af-4ed9-8b51-9527d55ee6da-dshm\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:28:21.421451 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.421421 2565 generic.go:356] "Generic (PLEG): container finished" podID="dc39b4a5-08af-4ed9-8b51-9527d55ee6da" containerID="cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777" exitCode=0 Jul 07 22:28:21.421567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.421480 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" Jul 07 22:28:21.421567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.421495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" event={"ID":"dc39b4a5-08af-4ed9-8b51-9527d55ee6da","Type":"ContainerDied","Data":"cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777"} Jul 07 22:28:21.421567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.421540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l" event={"ID":"dc39b4a5-08af-4ed9-8b51-9527d55ee6da","Type":"ContainerDied","Data":"0a5f7f906ee31491d7567af180f388b3eba130a2165c3fcce7d6beec1364dc3f"} Jul 07 22:28:21.421567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.421563 2565 scope.go:117] "RemoveContainer" containerID="cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777" Jul 07 22:28:21.431171 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.431148 2565 scope.go:117] "RemoveContainer" containerID="cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777" Jul 07 22:28:21.431520 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:28:21.431494 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777\": container with ID starting with cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777 not found: ID does not exist" containerID="cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777" Jul 07 22:28:21.431636 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.431528 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777"} err="failed to get container status \"cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777\": rpc error: code = NotFound desc = could not find container \"cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777\": container with ID starting with cb8b39019afdbd55662250e15f73dad01fe1621c4ce43bda05e3e96c7c91a777 not found: ID does not exist" Jul 07 22:28:21.440871 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.440834 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l"] Jul 07 22:28:21.444232 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:21.444207 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5ebf133b/e2e-lifecycle-model-5ebf133b-kserve-6567c4d5d7-d7w9l"] Jul 07 22:28:22.576574 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:22.576534 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc39b4a5-08af-4ed9-8b51-9527d55ee6da" path="/var/lib/kubelet/pods/dc39b4a5-08af-4ed9-8b51-9527d55ee6da/volumes" Jul 07 22:28:26.953718 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:26.953685 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:27.001283 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:27.001251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:38.938110 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:38.938076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:42.042007 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:42.041973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:43.011461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:43.011420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:48.651338 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:48.651297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:48.751433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:48.751397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:48.917873 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:48.917792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:50.216342 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:50.216305 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:50.315194 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:50.315155 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:50.416250 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:50.416216 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:51.118294 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:51.118254 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:51.222443 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:51.222387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:51.318326 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:51.318289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:52.689008 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:52.688973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:58.321368 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:58.321269 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:58.421526 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:58.421486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:28:59.021600 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:28:59.021563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:04.824046 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:04.823976 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:04.885256 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:04.885226 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:04.957172 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:04.957013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:05.057296 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:05.057259 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:15.547339 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:15.547304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:15.881921 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:15.881833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:36.858481 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:36.858441 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:42.564052 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:42.563991 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:42.664481 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:42.664449 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:42.913179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:42.913084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:43.267493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:43.267462 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:43.465324 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:43.465293 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:45.734874 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.734831 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:45.814997 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.814961 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl"] Jul 07 22:29:45.815407 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.815392 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc39b4a5-08af-4ed9-8b51-9527d55ee6da" containerName="main" Jul 07 22:29:45.815455 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.815409 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc39b4a5-08af-4ed9-8b51-9527d55ee6da" containerName="main" Jul 07 22:29:45.815455 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.815419 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b44c207-2889-4d2d-b32e-88a6650fa37a" containerName="maas-api" Jul 07 22:29:45.815455 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.815424 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b44c207-2889-4d2d-b32e-88a6650fa37a" containerName="maas-api" Jul 07 22:29:45.815580 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.815480 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc39b4a5-08af-4ed9-8b51-9527d55ee6da" containerName="main" Jul 07 22:29:45.815580 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.815488 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b44c207-2889-4d2d-b32e-88a6650fa37a" containerName="maas-api" Jul 07 22:29:45.818932 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.818910 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:45.821400 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.821369 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:29:45.821400 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.821255 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cd9893-serving-cert\"" Jul 07 22:29:45.831301 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.831270 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl"] Jul 07 22:29:45.833915 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.833883 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:45.891420 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.891386 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:45.917696 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.917665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fpn\" (UniqueName: \"kubernetes.io/projected/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-kube-api-access-f6fpn\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:45.917817 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.917709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:45.917817 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.917782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-maas-api-tls\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:45.992150 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:45.992017 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:46.018420 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.018383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fpn\" (UniqueName: \"kubernetes.io/projected/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-kube-api-access-f6fpn\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.018627 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.018428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.018627 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.018487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-maas-api-tls\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.019215 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.019188 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.021011 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.020989 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-maas-api-tls\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.025951 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.025928 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fpn\" (UniqueName: \"kubernetes.io/projected/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-kube-api-access-f6fpn\") pod \"maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.138082 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.138015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:46.285797 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.285760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl"] Jul 07 22:29:46.291317 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:29:46.291287 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d517fe1_ea3f_4d41_908a_b2312cb0fe9b.slice/crio-90bc7721c7ce48851076b20a780c4e380dc0f0307fa4b7bd93ca8f376050c3db WatchSource:0}: Error finding container 90bc7721c7ce48851076b20a780c4e380dc0f0307fa4b7bd93ca8f376050c3db: Status 404 returned error can't find the container with id 90bc7721c7ce48851076b20a780c4e380dc0f0307fa4b7bd93ca8f376050c3db Jul 07 22:29:46.292552 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.292534 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:29:46.296355 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.296329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:46.396256 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.396230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:46.844393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:46.844358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" event={"ID":"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b","Type":"ContainerStarted","Data":"90bc7721c7ce48851076b20a780c4e380dc0f0307fa4b7bd93ca8f376050c3db"} Jul 07 22:29:47.851541 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:47.851501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" event={"ID":"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b","Type":"ContainerStarted","Data":"ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc"} Jul 07 22:29:47.851975 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:47.851575 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:47.867720 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:47.867667 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" podStartSLOduration=2.374556709 podStartE2EDuration="2.867648837s" totalImagesPullingTime="493.092128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:29:47.866613346 +0000 UTC m=+1843.858186575" watchObservedRunningTime="2026-07-07 22:29:47.867648837 +0000 UTC m=+1843.859222065" Jul 07 22:29:50.250904 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:50.250865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:50.615100 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:50.614991 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:50.886561 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:50.886469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:51.019391 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:51.019349 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:53.757054 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.756983 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:53.851264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.851223 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:53.863002 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.862972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:29:53.915680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.915645 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf"] Jul 07 22:29:53.919417 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.919392 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:53.921301 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.921276 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fe6b32-serving-cert\"" Jul 07 22:29:53.927368 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.927328 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf"] Jul 07 22:29:53.975544 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.975509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:53.994379 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.994343 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4629dd0-579c-4c2f-8d77-50957406c243-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:53.994553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.994391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6jm\" (UniqueName: \"kubernetes.io/projected/c4629dd0-579c-4c2f-8d77-50957406c243-kube-api-access-dm6jm\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:53.994553 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:53.994420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4629dd0-579c-4c2f-8d77-50957406c243-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.076044 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.075939 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:54.095352 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.095315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4629dd0-579c-4c2f-8d77-50957406c243-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.095466 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.095366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6jm\" (UniqueName: \"kubernetes.io/projected/c4629dd0-579c-4c2f-8d77-50957406c243-kube-api-access-dm6jm\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.095466 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.095399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4629dd0-579c-4c2f-8d77-50957406c243-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.096135 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.096113 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4629dd0-579c-4c2f-8d77-50957406c243-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.098003 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.097983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4629dd0-579c-4c2f-8d77-50957406c243-maas-api-tls\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.103291 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.103260 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6jm\" (UniqueName: \"kubernetes.io/projected/c4629dd0-579c-4c2f-8d77-50957406c243-kube-api-access-dm6jm\") pod \"maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.233626 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.233590 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:54.388199 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.388109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf"] Jul 07 22:29:54.391630 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:29:54.391561 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4629dd0_579c_4c2f_8d77_50957406c243.slice/crio-884df2dd3df1ee8ece08036cfb28fb936602c755796968f5e51a428133e82b96 WatchSource:0}: Error finding container 884df2dd3df1ee8ece08036cfb28fb936602c755796968f5e51a428133e82b96: Status 404 returned error can't find the container with id 884df2dd3df1ee8ece08036cfb28fb936602c755796968f5e51a428133e82b96 Jul 07 22:29:54.408198 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.408160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:54.508544 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.508505 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:29:54.888244 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:54.888207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" event={"ID":"c4629dd0-579c-4c2f-8d77-50957406c243","Type":"ContainerStarted","Data":"884df2dd3df1ee8ece08036cfb28fb936602c755796968f5e51a428133e82b96"} Jul 07 22:29:55.895090 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:55.895051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" event={"ID":"c4629dd0-579c-4c2f-8d77-50957406c243","Type":"ContainerStarted","Data":"ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c"} Jul 07 22:29:55.895501 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:55.895115 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:29:55.910993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:55.910935 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" podStartSLOduration=2.446826716 podStartE2EDuration="2.910918173s" totalImagesPullingTime="464.091457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:29:55.90888017 +0000 UTC m=+1851.900453399" watchObservedRunningTime="2026-07-07 22:29:55.910918173 +0000 UTC m=+1851.902491401" Jul 07 22:29:58.256165 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:29:58.256119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:00.133421 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.133385 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r"] Jul 07 22:30:00.140340 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.140311 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm"] Jul 07 22:30:00.140502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.140410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:30:00.143888 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.143794 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr"] Jul 07 22:30:00.145057 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.144854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:30:00.148214 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.148183 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r"] Jul 07 22:30:00.148363 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.148297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:30:00.149966 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.149936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr"] Jul 07 22:30:00.152768 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.152739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm"] Jul 07 22:30:00.258960 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.258919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njr89\" (UniqueName: \"kubernetes.io/projected/8a19dd60-9257-473a-a3e4-bf3034709ea4-kube-api-access-njr89\") pod \"maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr\" (UID: \"8a19dd60-9257-473a-a3e4-bf3034709ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:30:00.258960 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.258959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfv9q\" (UniqueName: \"kubernetes.io/projected/b9118eac-0d04-498c-9372-6efed185d0ef-kube-api-access-kfv9q\") pod \"maas-api-key-cleanup-29724390-4jp5r\" (UID: \"b9118eac-0d04-498c-9372-6efed185d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:30:00.259231 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.259070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kffv\" (UniqueName: \"kubernetes.io/projected/383e1602-5721-49ea-a155-b4352709e873-kube-api-access-6kffv\") pod \"maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm\" (UID: \"383e1602-5721-49ea-a155-b4352709e873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:30:00.359956 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.359915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njr89\" (UniqueName: \"kubernetes.io/projected/8a19dd60-9257-473a-a3e4-bf3034709ea4-kube-api-access-njr89\") pod \"maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr\" (UID: \"8a19dd60-9257-473a-a3e4-bf3034709ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:30:00.359956 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.359956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfv9q\" (UniqueName: \"kubernetes.io/projected/b9118eac-0d04-498c-9372-6efed185d0ef-kube-api-access-kfv9q\") pod \"maas-api-key-cleanup-29724390-4jp5r\" (UID: \"b9118eac-0d04-498c-9372-6efed185d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:30:00.360247 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.360001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kffv\" (UniqueName: \"kubernetes.io/projected/383e1602-5721-49ea-a155-b4352709e873-kube-api-access-6kffv\") pod \"maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm\" (UID: \"383e1602-5721-49ea-a155-b4352709e873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:30:00.368931 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.368892 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfv9q\" (UniqueName: \"kubernetes.io/projected/b9118eac-0d04-498c-9372-6efed185d0ef-kube-api-access-kfv9q\") pod \"maas-api-key-cleanup-29724390-4jp5r\" (UID: \"b9118eac-0d04-498c-9372-6efed185d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:30:00.369081 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.368891 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njr89\" (UniqueName: \"kubernetes.io/projected/8a19dd60-9257-473a-a3e4-bf3034709ea4-kube-api-access-njr89\") pod \"maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr\" (UID: \"8a19dd60-9257-473a-a3e4-bf3034709ea4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:30:00.369081 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.368892 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kffv\" (UniqueName: \"kubernetes.io/projected/383e1602-5721-49ea-a155-b4352709e873-kube-api-access-6kffv\") pod \"maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm\" (UID: \"383e1602-5721-49ea-a155-b4352709e873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:30:00.456957 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.456856 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:30:00.465065 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.465041 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:30:00.472233 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.472202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:30:00.624770 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.624742 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r"] Jul 07 22:30:00.628101 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:30:00.628016 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9118eac_0d04_498c_9372_6efed185d0ef.slice/crio-e8ab205ebd969a7593d59ecd25b50a060d82ce6efcd56cc083667ad7f6da37ca WatchSource:0}: Error finding container e8ab205ebd969a7593d59ecd25b50a060d82ce6efcd56cc083667ad7f6da37ca: Status 404 returned error can't find the container with id e8ab205ebd969a7593d59ecd25b50a060d82ce6efcd56cc083667ad7f6da37ca Jul 07 22:30:00.653779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.653747 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm"] Jul 07 22:30:00.655078 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:30:00.655046 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383e1602_5721_49ea_a155_b4352709e873.slice/crio-9ed545d147927d2254a64471e938c8f13fa32b47f5eb1fdccc73ded7ec521feb WatchSource:0}: Error finding container 9ed545d147927d2254a64471e938c8f13fa32b47f5eb1fdccc73ded7ec521feb: Status 404 returned error can't find the container with id 9ed545d147927d2254a64471e938c8f13fa32b47f5eb1fdccc73ded7ec521feb Jul 07 22:30:00.684815 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.684769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr"] Jul 07 22:30:00.687277 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:30:00.687251 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a19dd60_9257_473a_a3e4_bf3034709ea4.slice/crio-aba142d1c6f08dd890eed1af2c658de8017692f75446cdadc16434f42e35e94b WatchSource:0}: Error finding container aba142d1c6f08dd890eed1af2c658de8017692f75446cdadc16434f42e35e94b: Status 404 returned error can't find the container with id aba142d1c6f08dd890eed1af2c658de8017692f75446cdadc16434f42e35e94b Jul 07 22:30:00.921351 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.921311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" event={"ID":"8a19dd60-9257-473a-a3e4-bf3034709ea4","Type":"ContainerStarted","Data":"aba142d1c6f08dd890eed1af2c658de8017692f75446cdadc16434f42e35e94b"} Jul 07 22:30:00.922608 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.922582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" event={"ID":"383e1602-5721-49ea-a155-b4352709e873","Type":"ContainerStarted","Data":"9ed545d147927d2254a64471e938c8f13fa32b47f5eb1fdccc73ded7ec521feb"} Jul 07 22:30:00.923769 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:00.923732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" event={"ID":"b9118eac-0d04-498c-9372-6efed185d0ef","Type":"ContainerStarted","Data":"e8ab205ebd969a7593d59ecd25b50a060d82ce6efcd56cc083667ad7f6da37ca"} Jul 07 22:30:01.905181 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.905140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:30:01.931916 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.931872 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" event={"ID":"8a19dd60-9257-473a-a3e4-bf3034709ea4","Type":"ContainerStarted","Data":"0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2"} Jul 07 22:30:01.933712 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.933680 2565 generic.go:356] "Generic (PLEG): container finished" podID="383e1602-5721-49ea-a155-b4352709e873" containerID="4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388" exitCode=7 Jul 07 22:30:01.933902 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.933761 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" event={"ID":"383e1602-5721-49ea-a155-b4352709e873","Type":"ContainerDied","Data":"4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388"} Jul 07 22:30:01.933966 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.933941 2565 scope.go:117] "RemoveContainer" containerID="4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388" Jul 07 22:30:01.936096 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.936069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" event={"ID":"b9118eac-0d04-498c-9372-6efed185d0ef","Type":"ContainerStarted","Data":"8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5"} Jul 07 22:30:01.953990 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.953930 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" podStartSLOduration=1.953908207 podStartE2EDuration="1.953908207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:30:01.951418358 +0000 UTC m=+1857.942991586" watchObservedRunningTime="2026-07-07 22:30:01.953908207 +0000 UTC m=+1857.945481440" Jul 07 22:30:01.983799 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:01.983736 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" podStartSLOduration=1.983718482 podStartE2EDuration="1.983718482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:30:01.982614097 +0000 UTC m=+1857.974187347" watchObservedRunningTime="2026-07-07 22:30:01.983718482 +0000 UTC m=+1857.975291711" Jul 07 22:30:02.943042 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:02.942991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" event={"ID":"383e1602-5721-49ea-a155-b4352709e873","Type":"ContainerStarted","Data":"b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45"} Jul 07 22:30:02.959535 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:02.959476 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" podStartSLOduration=2.959458197 podStartE2EDuration="2.959458197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:30:02.957974826 +0000 UTC m=+1858.949548054" watchObservedRunningTime="2026-07-07 22:30:02.959458197 +0000 UTC m=+1858.951031425" Jul 07 22:30:07.023651 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:07.023602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:07.124626 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:07.124564 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:09.365545 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.365495 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4"] Jul 07 22:30:09.370243 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.370217 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.373330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.373261 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cd9893\"/\"default-dockercfg-qr5s8\"" Jul 07 22:30:09.373330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.373309 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cd9893\"/\"kube-root-ca.crt\"" Jul 07 22:30:09.373587 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.373327 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cd9893\"/\"test-model-cd9893-kserve-self-signed-certs\"" Jul 07 22:30:09.373647 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.373593 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cd9893\"/\"openshift-service-ca.crt\"" Jul 07 22:30:09.376200 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.376169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4"] Jul 07 22:30:09.381997 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.381969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:09.452082 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.452036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-home\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.452271 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.452121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tmp-dir\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.452271 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.452154 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-model-cache\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.452271 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.452196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-dshm\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.452447 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.452331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pqd\" (UniqueName: \"kubernetes.io/projected/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-kube-api-access-p4pqd\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.452447 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.452398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tls-certs\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.553812 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.553759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tls-certs\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554012 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.553843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-home\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554012 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.553869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tmp-dir\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554012 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.553887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-model-cache\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554196 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.554007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-dshm\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554196 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.554120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pqd\" (UniqueName: \"kubernetes.io/projected/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-kube-api-access-p4pqd\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554370 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.554331 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tmp-dir\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554370 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.554353 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-model-cache\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.554495 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.554398 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-home\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.556477 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.556448 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-dshm\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.556608 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.556590 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tls-certs\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.562484 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.562459 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pqd\" (UniqueName: \"kubernetes.io/projected/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-kube-api-access-p4pqd\") pod \"test-model-cd9893-kserve-6c86fb5985-qb7j4\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.681660 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.681547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:09.828393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.828367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4"] Jul 07 22:30:09.830859 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:30:09.830836 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7984eaa0_87b6_4a26_8c4a_ab907ed48a43.slice/crio-d8606a09f82b09b611a2b4f9dbbe779c4cd7b698bd6d224f83c7aa9fabfd5e87 WatchSource:0}: Error finding container d8606a09f82b09b611a2b4f9dbbe779c4cd7b698bd6d224f83c7aa9fabfd5e87: Status 404 returned error can't find the container with id d8606a09f82b09b611a2b4f9dbbe779c4cd7b698bd6d224f83c7aa9fabfd5e87 Jul 07 22:30:09.981851 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.981750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" event={"ID":"7984eaa0-87b6-4a26-8c4a-ab907ed48a43","Type":"ContainerStarted","Data":"d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2"} Jul 07 22:30:09.981851 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.981794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" event={"ID":"7984eaa0-87b6-4a26-8c4a-ab907ed48a43","Type":"ContainerStarted","Data":"d8606a09f82b09b611a2b4f9dbbe779c4cd7b698bd6d224f83c7aa9fabfd5e87"} Jul 07 22:30:09.998982 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:09.998927 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" podStartSLOduration=0.998907889 podStartE2EDuration="998.907889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:30:09.995780714 +0000 UTC m=+1865.987353944" watchObservedRunningTime="2026-07-07 22:30:09.998907889 +0000 UTC m=+1865.990481118" Jul 07 22:30:15.889398 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:15.889356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:15.990317 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:15.990279 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:19.682685 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:19.682639 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:19.683131 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:19.682714 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:19.695726 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:19.695699 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:20.046203 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:20.046158 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:30:30.051222 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.051130 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz"] Jul 07 22:30:30.055635 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.055592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.058505 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.058477 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe6b32\"/\"default-dockercfg-9frfg\"" Jul 07 22:30:30.058505 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.058496 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe6b32\"/\"openshift-service-ca.crt\"" Jul 07 22:30:30.058747 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.058518 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe6b32\"/\"kube-root-ca.crt\"" Jul 07 22:30:30.058747 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.058500 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe6b32\"/\"test-model-fe6b32-kserve-self-signed-certs\"" Jul 07 22:30:30.063422 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.063391 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz"] Jul 07 22:30:30.073394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.073357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:30.154692 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.154641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-dshm\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.154915 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.154709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-model-cache\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.154915 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.154763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dr8\" (UniqueName: \"kubernetes.io/projected/57cf7b3a-e00f-4b60-95ee-0492554a3230-kube-api-access-g4dr8\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.154915 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.154799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-home\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.154915 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.154901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf7b3a-e00f-4b60-95ee-0492554a3230-tls-certs\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.155163 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.154937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-tmp-dir\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.255758 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.255678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-model-cache\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.255758 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.255766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dr8\" (UniqueName: \"kubernetes.io/projected/57cf7b3a-e00f-4b60-95ee-0492554a3230-kube-api-access-g4dr8\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.256086 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.255816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-home\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.256086 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.255872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf7b3a-e00f-4b60-95ee-0492554a3230-tls-certs\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.256086 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.255898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-tmp-dir\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.256086 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.255936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-dshm\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.256708 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.256666 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-model-cache\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.256876 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.256748 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-tmp-dir\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.257462 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.257437 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-home\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.259358 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.259328 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-dshm\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.265450 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.263980 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dr8\" (UniqueName: \"kubernetes.io/projected/57cf7b3a-e00f-4b60-95ee-0492554a3230-kube-api-access-g4dr8\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.265450 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.264058 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf7b3a-e00f-4b60-95ee-0492554a3230-tls-certs\") pod \"test-model-fe6b32-kserve-6c55f646fc-w52vz\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.367694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.367594 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:30.516303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:30.516246 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz"] Jul 07 22:30:30.519702 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:30:30.519669 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57cf7b3a_e00f_4b60_95ee_0492554a3230.slice/crio-bd52a975857e1fa0e93f3fb98ba30b7d8b8b90586094ba23ee6dd6156551378b WatchSource:0}: Error finding container bd52a975857e1fa0e93f3fb98ba30b7d8b8b90586094ba23ee6dd6156551378b: Status 404 returned error can't find the container with id bd52a975857e1fa0e93f3fb98ba30b7d8b8b90586094ba23ee6dd6156551378b Jul 07 22:30:31.095428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:31.095387 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" event={"ID":"57cf7b3a-e00f-4b60-95ee-0492554a3230","Type":"ContainerStarted","Data":"60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370"} Jul 07 22:30:31.095428 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:31.095436 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" event={"ID":"57cf7b3a-e00f-4b60-95ee-0492554a3230","Type":"ContainerStarted","Data":"bd52a975857e1fa0e93f3fb98ba30b7d8b8b90586094ba23ee6dd6156551378b"} Jul 07 22:30:31.110552 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:31.110501 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" podStartSLOduration=1.110483342 podStartE2EDuration="1.110483342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:30:31.109617676 +0000 UTC m=+1887.101190896" watchObservedRunningTime="2026-07-07 22:30:31.110483342 +0000 UTC m=+1887.102056570" Jul 07 22:30:40.367828 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:40.367785 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:40.368383 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:40.367845 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:40.380740 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:40.380711 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:41.159700 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:41.159667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:30:51.550348 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:51.550305 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4"] Jul 07 22:30:51.571885 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:51.571848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:52.581433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:52.581399 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz"] Jul 07 22:30:52.965834 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:52.965742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:53.239619 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.239519 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:53.340114 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.340082 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:53.439527 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.439494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:53.527479 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.527443 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl"] Jul 07 22:30:53.527723 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.527698 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" podUID="8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" containerName="maas-api" containerID="cri-o://ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc" gracePeriod=30 Jul 07 22:30:53.554661 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.554629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:30:53.555803 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.555773 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr"] Jul 07 22:30:53.556069 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.555999 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" podUID="8a19dd60-9257-473a-a3e4-bf3034709ea4" containerName="cleanup" containerID="cri-o://0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2" gracePeriod=30 Jul 07 22:30:53.788437 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.788370 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:30:53.896365 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.896326 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-maas-api-tls\") pod \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " Jul 07 22:30:53.896566 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.896474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-openshift-service-ca\") pod \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " Jul 07 22:30:53.896566 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.896520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6fpn\" (UniqueName: \"kubernetes.io/projected/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-kube-api-access-f6fpn\") pod \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\" (UID: \"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b\") " Jul 07 22:30:53.896840 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.896815 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" (UID: "8d517fe1-ea3f-4d41-908a-b2312cb0fe9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:30:53.898750 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.898727 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" (UID: "8d517fe1-ea3f-4d41-908a-b2312cb0fe9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:30:53.898815 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.898766 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-kube-api-access-f6fpn" (OuterVolumeSpecName: "kube-api-access-f6fpn") pod "8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" (UID: "8d517fe1-ea3f-4d41-908a-b2312cb0fe9b"). InnerVolumeSpecName "kube-api-access-f6fpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:30:53.997514 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.997476 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:30:53.997514 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.997508 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fpn\" (UniqueName: \"kubernetes.io/projected/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-kube-api-access-f6fpn\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:30:53.997514 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:53.997521 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:30:54.220255 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.220153 2565 generic.go:356] "Generic (PLEG): container finished" podID="8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" containerID="ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc" exitCode=0 Jul 07 22:30:54.220255 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.220201 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" Jul 07 22:30:54.220255 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.220242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" event={"ID":"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b","Type":"ContainerDied","Data":"ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc"} Jul 07 22:30:54.220539 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.220303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl" event={"ID":"8d517fe1-ea3f-4d41-908a-b2312cb0fe9b","Type":"ContainerDied","Data":"90bc7721c7ce48851076b20a780c4e380dc0f0307fa4b7bd93ca8f376050c3db"} Jul 07 22:30:54.220539 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.220332 2565 scope.go:117] "RemoveContainer" containerID="ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc" Jul 07 22:30:54.229862 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.229837 2565 scope.go:117] "RemoveContainer" containerID="ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc" Jul 07 22:30:54.230217 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:30:54.230194 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc\": container with ID starting with ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc not found: ID does not exist" containerID="ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc" Jul 07 22:30:54.230291 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.230227 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc"} err="failed to get container status \"ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc\": rpc error: code = NotFound desc = could not find container \"ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc\": container with ID starting with ebf8b5bb3d53200ce764e2b546fd8948411e6e0710b29c471e9d4754816138cc not found: ID does not exist" Jul 07 22:30:54.240200 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.240167 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl"] Jul 07 22:30:54.243590 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.243562 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd9893-5655bdccf4-xmggl"] Jul 07 22:30:54.577038 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:54.576969 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" path="/var/lib/kubelet/pods/8d517fe1-ea3f-4d41-908a-b2312cb0fe9b/volumes" Jul 07 22:30:58.423115 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:30:58.423076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:00.010742 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:00.010704 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r"] Jul 07 22:31:00.011439 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:00.010917 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" podUID="b9118eac-0d04-498c-9372-6efed185d0ef" containerName="cleanup" containerID="cri-o://8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5" gracePeriod=30 Jul 07 22:31:00.013902 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:00.013861 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm"] Jul 07 22:31:00.014157 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:00.014115 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" containerID="cri-o://b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45" gracePeriod=30 Jul 07 22:31:06.572993 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.572937 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" podUID="7984eaa0-87b6-4a26-8c4a-ab907ed48a43" containerName="main" containerID="cri-o://d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2" gracePeriod=45 Jul 07 22:31:06.843581 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.843553 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:31:06.919926 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.919890 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-model-cache\") pod \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " Jul 07 22:31:06.919926 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.919934 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-home\") pod \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " Jul 07 22:31:06.920230 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.919955 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tls-certs\") pod \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " Jul 07 22:31:06.920230 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.919983 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-dshm\") pod \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " Jul 07 22:31:06.920230 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920053 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tmp-dir\") pod \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " Jul 07 22:31:06.920230 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4pqd\" (UniqueName: \"kubernetes.io/projected/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-kube-api-access-p4pqd\") pod \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\" (UID: \"7984eaa0-87b6-4a26-8c4a-ab907ed48a43\") " Jul 07 22:31:06.920447 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920255 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-home" (OuterVolumeSpecName: "home") pod "7984eaa0-87b6-4a26-8c4a-ab907ed48a43" (UID: "7984eaa0-87b6-4a26-8c4a-ab907ed48a43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:06.920447 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920261 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-model-cache" (OuterVolumeSpecName: "model-cache") pod "7984eaa0-87b6-4a26-8c4a-ab907ed48a43" (UID: "7984eaa0-87b6-4a26-8c4a-ab907ed48a43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:06.920530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920471 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7984eaa0-87b6-4a26-8c4a-ab907ed48a43" (UID: "7984eaa0-87b6-4a26-8c4a-ab907ed48a43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:06.920530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920501 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-model-cache\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:06.920530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.920519 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-home\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:06.922588 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.922554 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-dshm" (OuterVolumeSpecName: "dshm") pod "7984eaa0-87b6-4a26-8c4a-ab907ed48a43" (UID: "7984eaa0-87b6-4a26-8c4a-ab907ed48a43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:06.922722 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.922575 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-kube-api-access-p4pqd" (OuterVolumeSpecName: "kube-api-access-p4pqd") pod "7984eaa0-87b6-4a26-8c4a-ab907ed48a43" (UID: "7984eaa0-87b6-4a26-8c4a-ab907ed48a43"). InnerVolumeSpecName "kube-api-access-p4pqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:31:06.922722 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:06.922582 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7984eaa0-87b6-4a26-8c4a-ab907ed48a43" (UID: "7984eaa0-87b6-4a26-8c4a-ab907ed48a43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:31:07.021493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.021452 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tls-certs\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:07.021493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.021485 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-dshm\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:07.021701 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.021516 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-tmp-dir\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:07.021701 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.021525 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pqd\" (UniqueName: \"kubernetes.io/projected/7984eaa0-87b6-4a26-8c4a-ab907ed48a43-kube-api-access-p4pqd\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:07.290826 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.290791 2565 generic.go:356] "Generic (PLEG): container finished" podID="7984eaa0-87b6-4a26-8c4a-ab907ed48a43" containerID="d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2" exitCode=0 Jul 07 22:31:07.290998 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.290839 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" Jul 07 22:31:07.290998 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.290880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" event={"ID":"7984eaa0-87b6-4a26-8c4a-ab907ed48a43","Type":"ContainerDied","Data":"d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2"} Jul 07 22:31:07.290998 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.290931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4" event={"ID":"7984eaa0-87b6-4a26-8c4a-ab907ed48a43","Type":"ContainerDied","Data":"d8606a09f82b09b611a2b4f9dbbe779c4cd7b698bd6d224f83c7aa9fabfd5e87"} Jul 07 22:31:07.290998 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.290955 2565 scope.go:117] "RemoveContainer" containerID="d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2" Jul 07 22:31:07.300939 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.300920 2565 scope.go:117] "RemoveContainer" containerID="d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2" Jul 07 22:31:07.301278 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:07.301259 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2\": container with ID starting with d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2 not found: ID does not exist" containerID="d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2" Jul 07 22:31:07.301327 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.301289 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2"} err="failed to get container status \"d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2\": rpc error: code = NotFound desc = could not find container \"d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2\": container with ID starting with d6a69553088dfa5ffef707c64f6f2a92ea1952e8ef18c914ce98ddd10c891bf2 not found: ID does not exist" Jul 07 22:31:07.310722 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.310690 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4"] Jul 07 22:31:07.314239 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.314211 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cd9893/test-model-cd9893-kserve-6c86fb5985-qb7j4"] Jul 07 22:31:07.603202 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.603107 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" podUID="57cf7b3a-e00f-4b60-95ee-0492554a3230" containerName="main" containerID="cri-o://60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370" gracePeriod=45 Jul 07 22:31:07.870792 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.870764 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:31:07.929934 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.929893 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4dr8\" (UniqueName: \"kubernetes.io/projected/57cf7b3a-e00f-4b60-95ee-0492554a3230-kube-api-access-g4dr8\") pod \"57cf7b3a-e00f-4b60-95ee-0492554a3230\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " Jul 07 22:31:07.930166 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.929960 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-dshm\") pod \"57cf7b3a-e00f-4b60-95ee-0492554a3230\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " Jul 07 22:31:07.930166 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.929980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-tmp-dir\") pod \"57cf7b3a-e00f-4b60-95ee-0492554a3230\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " Jul 07 22:31:07.930166 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.930064 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-model-cache\") pod \"57cf7b3a-e00f-4b60-95ee-0492554a3230\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " Jul 07 22:31:07.930166 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.930082 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf7b3a-e00f-4b60-95ee-0492554a3230-tls-certs\") pod \"57cf7b3a-e00f-4b60-95ee-0492554a3230\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " Jul 07 22:31:07.930166 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.930133 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-home\") pod \"57cf7b3a-e00f-4b60-95ee-0492554a3230\" (UID: \"57cf7b3a-e00f-4b60-95ee-0492554a3230\") " Jul 07 22:31:07.930449 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.930362 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57cf7b3a-e00f-4b60-95ee-0492554a3230" (UID: "57cf7b3a-e00f-4b60-95ee-0492554a3230"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:07.930449 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.930373 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-model-cache" (OuterVolumeSpecName: "model-cache") pod "57cf7b3a-e00f-4b60-95ee-0492554a3230" (UID: "57cf7b3a-e00f-4b60-95ee-0492554a3230"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:07.930546 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.930502 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-home" (OuterVolumeSpecName: "home") pod "57cf7b3a-e00f-4b60-95ee-0492554a3230" (UID: "57cf7b3a-e00f-4b60-95ee-0492554a3230"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:07.932487 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.932456 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-dshm" (OuterVolumeSpecName: "dshm") pod "57cf7b3a-e00f-4b60-95ee-0492554a3230" (UID: "57cf7b3a-e00f-4b60-95ee-0492554a3230"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 22:31:07.932608 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.932506 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cf7b3a-e00f-4b60-95ee-0492554a3230-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57cf7b3a-e00f-4b60-95ee-0492554a3230" (UID: "57cf7b3a-e00f-4b60-95ee-0492554a3230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:31:07.932608 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:07.932539 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cf7b3a-e00f-4b60-95ee-0492554a3230-kube-api-access-g4dr8" (OuterVolumeSpecName: "kube-api-access-g4dr8") pod "57cf7b3a-e00f-4b60-95ee-0492554a3230" (UID: "57cf7b3a-e00f-4b60-95ee-0492554a3230"). InnerVolumeSpecName "kube-api-access-g4dr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:31:08.031887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.031839 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-home\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:08.031887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.031878 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4dr8\" (UniqueName: \"kubernetes.io/projected/57cf7b3a-e00f-4b60-95ee-0492554a3230-kube-api-access-g4dr8\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:08.031887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.031892 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-dshm\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:08.031887 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.031902 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-tmp-dir\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:08.032211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.031914 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf7b3a-e00f-4b60-95ee-0492554a3230-model-cache\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:08.032211 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.031924 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf7b3a-e00f-4b60-95ee-0492554a3230-tls-certs\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:08.298151 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.298116 2565 generic.go:356] "Generic (PLEG): container finished" podID="57cf7b3a-e00f-4b60-95ee-0492554a3230" containerID="60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370" exitCode=0 Jul 07 22:31:08.298369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.298166 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" Jul 07 22:31:08.298369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.298199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" event={"ID":"57cf7b3a-e00f-4b60-95ee-0492554a3230","Type":"ContainerDied","Data":"60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370"} Jul 07 22:31:08.298369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.298236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz" event={"ID":"57cf7b3a-e00f-4b60-95ee-0492554a3230","Type":"ContainerDied","Data":"bd52a975857e1fa0e93f3fb98ba30b7d8b8b90586094ba23ee6dd6156551378b"} Jul 07 22:31:08.298369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.298255 2565 scope.go:117] "RemoveContainer" containerID="60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370" Jul 07 22:31:08.308678 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.308656 2565 scope.go:117] "RemoveContainer" containerID="60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370" Jul 07 22:31:08.308962 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:08.308946 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370\": container with ID starting with 60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370 not found: ID does not exist" containerID="60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370" Jul 07 22:31:08.309007 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.308970 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370"} err="failed to get container status \"60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370\": rpc error: code = NotFound desc = could not find container \"60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370\": container with ID starting with 60d9a676fade667c85baab752e4855ae9643c19d23d07c799d7267084f1c4370 not found: ID does not exist" Jul 07 22:31:08.318527 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.318424 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz"] Jul 07 22:31:08.320646 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.320617 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fe6b32/test-model-fe6b32-kserve-6c55f646fc-w52vz"] Jul 07 22:31:08.576642 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.576557 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cf7b3a-e00f-4b60-95ee-0492554a3230" path="/var/lib/kubelet/pods/57cf7b3a-e00f-4b60-95ee-0492554a3230/volumes" Jul 07 22:31:08.576965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:08.576951 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7984eaa0-87b6-4a26-8c4a-ab907ed48a43" path="/var/lib/kubelet/pods/7984eaa0-87b6-4a26-8c4a-ab907ed48a43/volumes" Jul 07 22:31:12.882680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:12.882631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:23.744294 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:23.744262 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:31:23.887593 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:23.887504 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njr89\" (UniqueName: \"kubernetes.io/projected/8a19dd60-9257-473a-a3e4-bf3034709ea4-kube-api-access-njr89\") pod \"8a19dd60-9257-473a-a3e4-bf3034709ea4\" (UID: \"8a19dd60-9257-473a-a3e4-bf3034709ea4\") " Jul 07 22:31:23.889812 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:23.889779 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a19dd60-9257-473a-a3e4-bf3034709ea4-kube-api-access-njr89" (OuterVolumeSpecName: "kube-api-access-njr89") pod "8a19dd60-9257-473a-a3e4-bf3034709ea4" (UID: "8a19dd60-9257-473a-a3e4-bf3034709ea4"). InnerVolumeSpecName "kube-api-access-njr89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:31:23.988618 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:23.988583 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njr89\" (UniqueName: \"kubernetes.io/projected/8a19dd60-9257-473a-a3e4-bf3034709ea4-kube-api-access-njr89\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:24.388118 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.388077 2565 generic.go:356] "Generic (PLEG): container finished" podID="8a19dd60-9257-473a-a3e4-bf3034709ea4" containerID="0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2" exitCode=137 Jul 07 22:31:24.388303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.388135 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" Jul 07 22:31:24.388303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.388157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" event={"ID":"8a19dd60-9257-473a-a3e4-bf3034709ea4","Type":"ContainerDied","Data":"0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2"} Jul 07 22:31:24.388303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.388236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr" event={"ID":"8a19dd60-9257-473a-a3e4-bf3034709ea4","Type":"ContainerDied","Data":"aba142d1c6f08dd890eed1af2c658de8017692f75446cdadc16434f42e35e94b"} Jul 07 22:31:24.388303 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.388260 2565 scope.go:117] "RemoveContainer" containerID="0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2" Jul 07 22:31:24.397567 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.397546 2565 scope.go:117] "RemoveContainer" containerID="0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2" Jul 07 22:31:24.397867 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:24.397848 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2\": container with ID starting with 0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2 not found: ID does not exist" containerID="0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2" Jul 07 22:31:24.397913 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.397876 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2"} err="failed to get container status \"0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2\": rpc error: code = NotFound desc = could not find container \"0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2\": container with ID starting with 0d768f5837ecbcfa1570a92cefdc5b96ba3b137de9c56d89c2615735f8876da2 not found: ID does not exist" Jul 07 22:31:24.408085 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.408051 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr"] Jul 07 22:31:24.411112 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.411085 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-cd9893-29724390-bccqr"] Jul 07 22:31:24.577687 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:24.577649 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a19dd60-9257-473a-a3e4-bf3034709ea4" path="/var/lib/kubelet/pods/8a19dd60-9257-473a-a3e4-bf3034709ea4/volumes" Jul 07 22:31:30.202501 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.202470 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:31:30.238875 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.238843 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:31:30.353142 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.353009 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kffv\" (UniqueName: \"kubernetes.io/projected/383e1602-5721-49ea-a155-b4352709e873-kube-api-access-6kffv\") pod \"383e1602-5721-49ea-a155-b4352709e873\" (UID: \"383e1602-5721-49ea-a155-b4352709e873\") " Jul 07 22:31:30.353142 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.353131 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfv9q\" (UniqueName: \"kubernetes.io/projected/b9118eac-0d04-498c-9372-6efed185d0ef-kube-api-access-kfv9q\") pod \"b9118eac-0d04-498c-9372-6efed185d0ef\" (UID: \"b9118eac-0d04-498c-9372-6efed185d0ef\") " Jul 07 22:31:30.355377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.355341 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383e1602-5721-49ea-a155-b4352709e873-kube-api-access-6kffv" (OuterVolumeSpecName: "kube-api-access-6kffv") pod "383e1602-5721-49ea-a155-b4352709e873" (UID: "383e1602-5721-49ea-a155-b4352709e873"). InnerVolumeSpecName "kube-api-access-6kffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:31:30.355377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.355364 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9118eac-0d04-498c-9372-6efed185d0ef-kube-api-access-kfv9q" (OuterVolumeSpecName: "kube-api-access-kfv9q") pod "b9118eac-0d04-498c-9372-6efed185d0ef" (UID: "b9118eac-0d04-498c-9372-6efed185d0ef"). InnerVolumeSpecName "kube-api-access-kfv9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:31:30.420819 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.420780 2565 generic.go:356] "Generic (PLEG): container finished" podID="383e1602-5721-49ea-a155-b4352709e873" containerID="b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45" exitCode=137 Jul 07 22:31:30.421041 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.420833 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" Jul 07 22:31:30.421041 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.420864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" event={"ID":"383e1602-5721-49ea-a155-b4352709e873","Type":"ContainerDied","Data":"b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45"} Jul 07 22:31:30.421041 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.420914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm" event={"ID":"383e1602-5721-49ea-a155-b4352709e873","Type":"ContainerDied","Data":"9ed545d147927d2254a64471e938c8f13fa32b47f5eb1fdccc73ded7ec521feb"} Jul 07 22:31:30.421041 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.420937 2565 scope.go:117] "RemoveContainer" containerID="b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45" Jul 07 22:31:30.422548 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.422524 2565 generic.go:356] "Generic (PLEG): container finished" podID="b9118eac-0d04-498c-9372-6efed185d0ef" containerID="8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5" exitCode=137 Jul 07 22:31:30.422674 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.422585 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" Jul 07 22:31:30.422674 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.422583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" event={"ID":"b9118eac-0d04-498c-9372-6efed185d0ef","Type":"ContainerDied","Data":"8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5"} Jul 07 22:31:30.422674 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.422652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r" event={"ID":"b9118eac-0d04-498c-9372-6efed185d0ef","Type":"ContainerDied","Data":"e8ab205ebd969a7593d59ecd25b50a060d82ce6efcd56cc083667ad7f6da37ca"} Jul 07 22:31:30.437509 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.437488 2565 scope.go:117] "RemoveContainer" containerID="4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388" Jul 07 22:31:30.446929 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.446897 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm"] Jul 07 22:31:30.450587 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.450562 2565 scope.go:117] "RemoveContainer" containerID="b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45" Jul 07 22:31:30.451204 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:30.451143 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45\": container with ID starting with b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45 not found: ID does not exist" containerID="b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45" Jul 07 22:31:30.451342 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.451194 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45"} err="failed to get container status \"b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45\": rpc error: code = NotFound desc = could not find container \"b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45\": container with ID starting with b86bcdcc8c844249d5706dbc8dee2591566a2fe034ef239ad5fd2006f8059b45 not found: ID does not exist" Jul 07 22:31:30.451342 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.451233 2565 scope.go:117] "RemoveContainer" containerID="4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388" Jul 07 22:31:30.451512 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.451492 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe6b32-29724390-6rbdm"] Jul 07 22:31:30.451581 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:30.451511 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388\": container with ID starting with 4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388 not found: ID does not exist" containerID="4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388" Jul 07 22:31:30.451581 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.451534 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388"} err="failed to get container status \"4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388\": rpc error: code = NotFound desc = could not find container \"4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388\": container with ID starting with 4dd4413f77ba23ace48c921de4c60bedeb402ef37c057e76dc7edfddfd57c388 not found: ID does not exist" Jul 07 22:31:30.451581 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.451551 2565 scope.go:117] "RemoveContainer" containerID="8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5" Jul 07 22:31:30.454299 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.454272 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kffv\" (UniqueName: \"kubernetes.io/projected/383e1602-5721-49ea-a155-b4352709e873-kube-api-access-6kffv\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:30.454299 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.454298 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfv9q\" (UniqueName: \"kubernetes.io/projected/b9118eac-0d04-498c-9372-6efed185d0ef-kube-api-access-kfv9q\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:30.461592 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.461572 2565 scope.go:117] "RemoveContainer" containerID="8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5" Jul 07 22:31:30.461780 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.461744 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r"] Jul 07 22:31:30.461976 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:30.461951 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5\": container with ID starting with 8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5 not found: ID does not exist" containerID="8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5" Jul 07 22:31:30.462101 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.461982 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5"} err="failed to get container status \"8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5\": rpc error: code = NotFound desc = could not find container \"8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5\": container with ID starting with 8fbab554579d619f45f5fbffed98dc74ae17360c34232c2ccbca6be8b62506c5 not found: ID does not exist" Jul 07 22:31:30.463740 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.463720 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29724390-4jp5r"] Jul 07 22:31:30.577305 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.577270 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383e1602-5721-49ea-a155-b4352709e873" path="/var/lib/kubelet/pods/383e1602-5721-49ea-a155-b4352709e873/volumes" Jul 07 22:31:30.577721 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:30.577700 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9118eac-0d04-498c-9372-6efed185d0ef" path="/var/lib/kubelet/pods/b9118eac-0d04-498c-9372-6efed185d0ef/volumes" Jul 07 22:31:35.208004 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:35.207959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:35.479497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:35.479383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:35.872577 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:35.872536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:36.292426 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:36.292393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:36.390827 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:36.390786 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:36.510167 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:36.510135 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf"] Jul 07 22:31:36.510408 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:36.510382 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" podUID="c4629dd0-579c-4c2f-8d77-50957406c243" containerName="maas-api" containerID="cri-o://ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c" gracePeriod=30 Jul 07 22:31:36.533128 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:36.533093 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:36.963302 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:36.963271 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:31:37.117464 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.117361 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm6jm\" (UniqueName: \"kubernetes.io/projected/c4629dd0-579c-4c2f-8d77-50957406c243-kube-api-access-dm6jm\") pod \"c4629dd0-579c-4c2f-8d77-50957406c243\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " Jul 07 22:31:37.117464 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.117418 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4629dd0-579c-4c2f-8d77-50957406c243-openshift-service-ca\") pod \"c4629dd0-579c-4c2f-8d77-50957406c243\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " Jul 07 22:31:37.117699 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.117493 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4629dd0-579c-4c2f-8d77-50957406c243-maas-api-tls\") pod \"c4629dd0-579c-4c2f-8d77-50957406c243\" (UID: \"c4629dd0-579c-4c2f-8d77-50957406c243\") " Jul 07 22:31:37.117756 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.117734 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4629dd0-579c-4c2f-8d77-50957406c243-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4629dd0-579c-4c2f-8d77-50957406c243" (UID: "c4629dd0-579c-4c2f-8d77-50957406c243"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:31:37.117848 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.117832 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4629dd0-579c-4c2f-8d77-50957406c243-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:37.119706 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.119682 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4629dd0-579c-4c2f-8d77-50957406c243-kube-api-access-dm6jm" (OuterVolumeSpecName: "kube-api-access-dm6jm") pod "c4629dd0-579c-4c2f-8d77-50957406c243" (UID: "c4629dd0-579c-4c2f-8d77-50957406c243"). InnerVolumeSpecName "kube-api-access-dm6jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:31:37.119706 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.119687 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4629dd0-579c-4c2f-8d77-50957406c243-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4629dd0-579c-4c2f-8d77-50957406c243" (UID: "c4629dd0-579c-4c2f-8d77-50957406c243"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:31:37.218957 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.218913 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm6jm\" (UniqueName: \"kubernetes.io/projected/c4629dd0-579c-4c2f-8d77-50957406c243-kube-api-access-dm6jm\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:37.218957 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.218948 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4629dd0-579c-4c2f-8d77-50957406c243-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:31:37.463456 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.463367 2565 generic.go:356] "Generic (PLEG): container finished" podID="c4629dd0-579c-4c2f-8d77-50957406c243" containerID="ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c" exitCode=0 Jul 07 22:31:37.463456 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.463418 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" Jul 07 22:31:37.463916 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.463456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" event={"ID":"c4629dd0-579c-4c2f-8d77-50957406c243","Type":"ContainerDied","Data":"ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c"} Jul 07 22:31:37.463916 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.463515 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" event={"ID":"c4629dd0-579c-4c2f-8d77-50957406c243","Type":"ContainerDied","Data":"884df2dd3df1ee8ece08036cfb28fb936602c755796968f5e51a428133e82b96"} Jul 07 22:31:37.463916 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.463546 2565 scope.go:117] "RemoveContainer" containerID="ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c" Jul 07 22:31:37.473189 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.473169 2565 scope.go:117] "RemoveContainer" containerID="ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c" Jul 07 22:31:37.473564 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:37.473541 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c\": container with ID starting with ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c not found: ID does not exist" containerID="ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c" Jul 07 22:31:37.473652 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.473571 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c"} err="failed to get container status \"ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c\": rpc error: code = NotFound desc = could not find container \"ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c\": container with ID starting with ea8eeae41fa83afad0d63cd6d5ab5cfc698caac918de91844802902951a12b8c not found: ID does not exist" Jul 07 22:31:37.483236 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.483198 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf"] Jul 07 22:31:37.486493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:37.486465 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf"] Jul 07 22:31:38.576678 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:38.576641 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4629dd0-579c-4c2f-8d77-50957406c243" path="/var/lib/kubelet/pods/c4629dd0-579c-4c2f-8d77-50957406c243/volumes" Jul 07 22:31:39.900665 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:39.900618 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fe6b32-67cfdf8bf-t8vmf" podUID="c4629dd0-579c-4c2f-8d77-50957406c243" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 07 22:31:42.199813 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:42.199768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:42.300261 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:42.300220 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:42.468522 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:42.468409 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:43.043326 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:43.043261 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:43.354715 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:43.354597 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:43.434370 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:43.434328 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:46.292323 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.292254 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:46.392895 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.392865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:46.413289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413257 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw"] Jul 07 22:31:46.413675 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413662 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a19dd60-9257-473a-a3e4-bf3034709ea4" containerName="cleanup" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413677 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a19dd60-9257-473a-a3e4-bf3034709ea4" containerName="cleanup" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413687 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413693 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413706 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" containerName="maas-api" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413711 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" containerName="maas-api" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413721 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" Jul 07 22:31:46.413727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413727 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413733 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cf7b3a-e00f-4b60-95ee-0492554a3230" containerName="main" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413739 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cf7b3a-e00f-4b60-95ee-0492554a3230" containerName="main" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413748 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4629dd0-579c-4c2f-8d77-50957406c243" containerName="maas-api" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413753 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4629dd0-579c-4c2f-8d77-50957406c243" containerName="maas-api" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413759 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9118eac-0d04-498c-9372-6efed185d0ef" containerName="cleanup" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413764 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9118eac-0d04-498c-9372-6efed185d0ef" containerName="cleanup" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413774 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7984eaa0-87b6-4a26-8c4a-ab907ed48a43" containerName="main" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413779 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7984eaa0-87b6-4a26-8c4a-ab907ed48a43" containerName="main" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413833 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4629dd0-579c-4c2f-8d77-50957406c243" containerName="maas-api" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413842 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d517fe1-ea3f-4d41-908a-b2312cb0fe9b" containerName="maas-api" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413846 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9118eac-0d04-498c-9372-6efed185d0ef" containerName="cleanup" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413853 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7984eaa0-87b6-4a26-8c4a-ab907ed48a43" containerName="main" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413859 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a19dd60-9257-473a-a3e4-bf3034709ea4" containerName="cleanup" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413865 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" Jul 07 22:31:46.413965 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.413873 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cf7b3a-e00f-4b60-95ee-0492554a3230" containerName="main" Jul 07 22:31:46.418117 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.418095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.420252 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.420224 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fa42f6-serving-cert\"" Jul 07 22:31:46.420394 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.420264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:31:46.424315 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.424267 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw"] Jul 07 22:31:46.498410 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.498377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.498631 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.498481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sb9r\" (UniqueName: \"kubernetes.io/projected/985ba99d-7931-4860-97c2-e5fdc97c50bb-kube-api-access-5sb9r\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.498631 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.498516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/985ba99d-7931-4860-97c2-e5fdc97c50bb-openshift-service-ca\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.508793 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.508752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:46.599984 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.599890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.599984 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.599952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sb9r\" (UniqueName: \"kubernetes.io/projected/985ba99d-7931-4860-97c2-e5fdc97c50bb-kube-api-access-5sb9r\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.599984 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.599975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/985ba99d-7931-4860-97c2-e5fdc97c50bb-openshift-service-ca\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.600252 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:46.600072 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-serving-cert: secret "maas-api-e2e-api-a-fa42f6-serving-cert" not found Jul 07 22:31:46.600252 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:31:46.600137 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls podName:985ba99d-7931-4860-97c2-e5fdc97c50bb nodeName:}" failed. No retries permitted until 2026-07-07 22:31:47.100121787 +0000 UTC m=+1963.091694992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls") pod "maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" (UID: "985ba99d-7931-4860-97c2-e5fdc97c50bb") : secret "maas-api-e2e-api-a-fa42f6-serving-cert" not found Jul 07 22:31:46.600612 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.600593 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/985ba99d-7931-4860-97c2-e5fdc97c50bb-openshift-service-ca\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.610418 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.610392 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sb9r\" (UniqueName: \"kubernetes.io/projected/985ba99d-7931-4860-97c2-e5fdc97c50bb-kube-api-access-5sb9r\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:46.948295 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:46.948203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:47.048651 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:47.048617 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:47.104975 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:47.104934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:47.107562 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:47.107541 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls\") pod \"maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:47.330254 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:47.330208 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:47.486839 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:47.486813 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw"] Jul 07 22:31:47.489168 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:31:47.489133 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985ba99d_7931_4860_97c2_e5fdc97c50bb.slice/crio-92040f9a44b60b601924497ce66807f0835b07180fe73ad725af0af0489991f7 WatchSource:0}: Error finding container 92040f9a44b60b601924497ce66807f0835b07180fe73ad725af0af0489991f7: Status 404 returned error can't find the container with id 92040f9a44b60b601924497ce66807f0835b07180fe73ad725af0af0489991f7 Jul 07 22:31:47.518868 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:47.518810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" event={"ID":"985ba99d-7931-4860-97c2-e5fdc97c50bb","Type":"ContainerStarted","Data":"92040f9a44b60b601924497ce66807f0835b07180fe73ad725af0af0489991f7"} Jul 07 22:31:48.524936 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:48.524890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" event={"ID":"985ba99d-7931-4860-97c2-e5fdc97c50bb","Type":"ContainerStarted","Data":"d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725"} Jul 07 22:31:48.525444 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:48.524991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:48.541396 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:48.541338 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" podStartSLOduration=1.9957249209999999 podStartE2EDuration="2.541321283s" totalImagesPullingTime="545.596362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:31:48.539850713 +0000 UTC m=+1964.531423941" watchObservedRunningTime="2026-07-07 22:31:48.541321283 +0000 UTC m=+1964.532894511" Jul 07 22:31:50.805151 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:50.805076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:51.262433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:51.262397 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:51.562153 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:51.562061 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:52.198960 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:52.198925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:54.344651 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.344615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:54.406367 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.406330 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc"] Jul 07 22:31:54.406856 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.406840 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="383e1602-5721-49ea-a155-b4352709e873" containerName="cleanup" Jul 07 22:31:54.417229 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.417186 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.419171 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.419141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-226f1f-serving-cert\"" Jul 07 22:31:54.420148 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.420119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc"] Jul 07 22:31:54.425900 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.425814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:54.476987 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.476888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fbf3578-be4c-406a-af40-c091891241ec-maas-api-tls\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.476987 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.476950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vbz\" (UniqueName: \"kubernetes.io/projected/5fbf3578-be4c-406a-af40-c091891241ec-kube-api-access-77vbz\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.477253 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.477073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbf3578-be4c-406a-af40-c091891241ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.490241 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.490204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:54.536156 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.536123 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:31:54.578636 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.578599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fbf3578-be4c-406a-af40-c091891241ec-maas-api-tls\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.578855 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.578728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vbz\" (UniqueName: \"kubernetes.io/projected/5fbf3578-be4c-406a-af40-c091891241ec-kube-api-access-77vbz\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.578855 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.578782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbf3578-be4c-406a-af40-c091891241ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.579594 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.579569 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbf3578-be4c-406a-af40-c091891241ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.581738 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.581715 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fbf3578-be4c-406a-af40-c091891241ec-maas-api-tls\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.586468 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.586434 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vbz\" (UniqueName: \"kubernetes.io/projected/5fbf3578-be4c-406a-af40-c091891241ec-kube-api-access-77vbz\") pod \"maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.653572 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.653458 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:54.731364 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.731320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:54.879952 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.879906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:54.880472 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.880449 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc"] Jul 07 22:31:54.884090 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:31:54.884062 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbf3578_be4c_406a_af40_c091891241ec.slice/crio-209f5bea6d625e06fd6fe989920556202d81b5a7aa950ce070eba4b12513c48e WatchSource:0}: Error finding container 209f5bea6d625e06fd6fe989920556202d81b5a7aa950ce070eba4b12513c48e: Status 404 returned error can't find the container with id 209f5bea6d625e06fd6fe989920556202d81b5a7aa950ce070eba4b12513c48e Jul 07 22:31:54.978680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:54.978578 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:31:55.564982 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:55.564879 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" event={"ID":"5fbf3578-be4c-406a-af40-c091891241ec","Type":"ContainerStarted","Data":"35c0af4d3793caf28d97c6b9afadf1c854104eb5a681f239f8c5cad8768b7bb6"} Jul 07 22:31:55.564982 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:55.564938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" event={"ID":"5fbf3578-be4c-406a-af40-c091891241ec","Type":"ContainerStarted","Data":"209f5bea6d625e06fd6fe989920556202d81b5a7aa950ce070eba4b12513c48e"} Jul 07 22:31:55.564982 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:55.564969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:31:55.582355 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:55.582294 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" podStartSLOduration=1.195977584 podStartE2EDuration="1.58227007s" totalImagesPullingTime="386.292486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:31:55.580178801 +0000 UTC m=+1971.571752030" watchObservedRunningTime="2026-07-07 22:31:55.58227007 +0000 UTC m=+1971.573843299" Jul 07 22:31:58.851078 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:31:58.851008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:01.575226 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:01.575191 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:32:07.094266 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.094224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:07.435480 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.435388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:07.537269 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.537235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:07.636067 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.636001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:07.750328 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.750251 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw"] Jul 07 22:32:07.750556 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.750525 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" podUID="985ba99d-7931-4860-97c2-e5fdc97c50bb" containerName="maas-api" containerID="cri-o://d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725" gracePeriod=30 Jul 07 22:32:07.773918 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:07.773888 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:08.011918 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.011843 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:32:08.114865 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.114829 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls\") pod \"985ba99d-7931-4860-97c2-e5fdc97c50bb\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " Jul 07 22:32:08.114865 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.114881 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sb9r\" (UniqueName: \"kubernetes.io/projected/985ba99d-7931-4860-97c2-e5fdc97c50bb-kube-api-access-5sb9r\") pod \"985ba99d-7931-4860-97c2-e5fdc97c50bb\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " Jul 07 22:32:08.115393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.114918 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/985ba99d-7931-4860-97c2-e5fdc97c50bb-openshift-service-ca\") pod \"985ba99d-7931-4860-97c2-e5fdc97c50bb\" (UID: \"985ba99d-7931-4860-97c2-e5fdc97c50bb\") " Jul 07 22:32:08.115393 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.115365 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985ba99d-7931-4860-97c2-e5fdc97c50bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "985ba99d-7931-4860-97c2-e5fdc97c50bb" (UID: "985ba99d-7931-4860-97c2-e5fdc97c50bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:32:08.117225 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.117191 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "985ba99d-7931-4860-97c2-e5fdc97c50bb" (UID: "985ba99d-7931-4860-97c2-e5fdc97c50bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:32:08.117349 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.117304 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985ba99d-7931-4860-97c2-e5fdc97c50bb-kube-api-access-5sb9r" (OuterVolumeSpecName: "kube-api-access-5sb9r") pod "985ba99d-7931-4860-97c2-e5fdc97c50bb" (UID: "985ba99d-7931-4860-97c2-e5fdc97c50bb"). InnerVolumeSpecName "kube-api-access-5sb9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:32:08.215991 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.215948 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/985ba99d-7931-4860-97c2-e5fdc97c50bb-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:32:08.215991 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.215986 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sb9r\" (UniqueName: \"kubernetes.io/projected/985ba99d-7931-4860-97c2-e5fdc97c50bb-kube-api-access-5sb9r\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:32:08.215991 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.216001 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/985ba99d-7931-4860-97c2-e5fdc97c50bb-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:32:08.629961 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.629927 2565 generic.go:356] "Generic (PLEG): container finished" podID="985ba99d-7931-4860-97c2-e5fdc97c50bb" containerID="d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725" exitCode=0 Jul 07 22:32:08.630182 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.629990 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" Jul 07 22:32:08.630182 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.630009 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" event={"ID":"985ba99d-7931-4860-97c2-e5fdc97c50bb","Type":"ContainerDied","Data":"d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725"} Jul 07 22:32:08.630182 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.630074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw" event={"ID":"985ba99d-7931-4860-97c2-e5fdc97c50bb","Type":"ContainerDied","Data":"92040f9a44b60b601924497ce66807f0835b07180fe73ad725af0af0489991f7"} Jul 07 22:32:08.630182 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.630098 2565 scope.go:117] "RemoveContainer" containerID="d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725" Jul 07 22:32:08.639733 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.639710 2565 scope.go:117] "RemoveContainer" containerID="d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725" Jul 07 22:32:08.640072 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:32:08.640042 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725\": container with ID starting with d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725 not found: ID does not exist" containerID="d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725" Jul 07 22:32:08.640145 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.640077 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725"} err="failed to get container status \"d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725\": rpc error: code = NotFound desc = could not find container \"d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725\": container with ID starting with d5eb17d39304a97890bb84341a382774668b6c6d127e6a855ac4a62a82040725 not found: ID does not exist" Jul 07 22:32:08.644745 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.644712 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw"] Jul 07 22:32:08.648947 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:08.648920 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa42f6-b5995c6cd-2n9gw"] Jul 07 22:32:10.577355 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:10.577319 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985ba99d-7931-4860-97c2-e5fdc97c50bb" path="/var/lib/kubelet/pods/985ba99d-7931-4860-97c2-e5fdc97c50bb/volumes" Jul 07 22:32:13.163831 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:13.163785 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:13.222308 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:13.222276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:13.417271 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:13.417188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:13.792801 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:13.792767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:14.195108 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.194975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:14.314426 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.314384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:14.414782 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.414748 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:14.525284 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.525250 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc"] Jul 07 22:32:14.525543 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.525517 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" podUID="5fbf3578-be4c-406a-af40-c091891241ec" containerName="maas-api" containerID="cri-o://35c0af4d3793caf28d97c6b9afadf1c854104eb5a681f239f8c5cad8768b7bb6" gracePeriod=30 Jul 07 22:32:14.549345 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.549312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:14.662055 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.662003 2565 generic.go:356] "Generic (PLEG): container finished" podID="5fbf3578-be4c-406a-af40-c091891241ec" containerID="35c0af4d3793caf28d97c6b9afadf1c854104eb5a681f239f8c5cad8768b7bb6" exitCode=0 Jul 07 22:32:14.662233 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.662059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" event={"ID":"5fbf3578-be4c-406a-af40-c091891241ec","Type":"ContainerDied","Data":"35c0af4d3793caf28d97c6b9afadf1c854104eb5a681f239f8c5cad8768b7bb6"} Jul 07 22:32:14.807947 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.807914 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:32:14.987300 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.987268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbf3578-be4c-406a-af40-c091891241ec-openshift-service-ca\") pod \"5fbf3578-be4c-406a-af40-c091891241ec\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " Jul 07 22:32:14.987499 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.987342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77vbz\" (UniqueName: \"kubernetes.io/projected/5fbf3578-be4c-406a-af40-c091891241ec-kube-api-access-77vbz\") pod \"5fbf3578-be4c-406a-af40-c091891241ec\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " Jul 07 22:32:14.987499 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.987368 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fbf3578-be4c-406a-af40-c091891241ec-maas-api-tls\") pod \"5fbf3578-be4c-406a-af40-c091891241ec\" (UID: \"5fbf3578-be4c-406a-af40-c091891241ec\") " Jul 07 22:32:14.987680 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.987634 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbf3578-be4c-406a-af40-c091891241ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fbf3578-be4c-406a-af40-c091891241ec" (UID: "5fbf3578-be4c-406a-af40-c091891241ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:32:14.989681 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.989651 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbf3578-be4c-406a-af40-c091891241ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fbf3578-be4c-406a-af40-c091891241ec" (UID: "5fbf3578-be4c-406a-af40-c091891241ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:32:14.989795 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:14.989651 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbf3578-be4c-406a-af40-c091891241ec-kube-api-access-77vbz" (OuterVolumeSpecName: "kube-api-access-77vbz") pod "5fbf3578-be4c-406a-af40-c091891241ec" (UID: "5fbf3578-be4c-406a-af40-c091891241ec"). InnerVolumeSpecName "kube-api-access-77vbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:32:15.088369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.088266 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vbz\" (UniqueName: \"kubernetes.io/projected/5fbf3578-be4c-406a-af40-c091891241ec-kube-api-access-77vbz\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:32:15.088369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.088299 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fbf3578-be4c-406a-af40-c091891241ec-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:32:15.088369 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.088310 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbf3578-be4c-406a-af40-c091891241ec-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:32:15.669009 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.668967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" event={"ID":"5fbf3578-be4c-406a-af40-c091891241ec","Type":"ContainerDied","Data":"209f5bea6d625e06fd6fe989920556202d81b5a7aa950ce070eba4b12513c48e"} Jul 07 22:32:15.669009 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.669002 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc" Jul 07 22:32:15.669497 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.669069 2565 scope.go:117] "RemoveContainer" containerID="35c0af4d3793caf28d97c6b9afadf1c854104eb5a681f239f8c5cad8768b7bb6" Jul 07 22:32:15.690034 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.689978 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc"] Jul 07 22:32:15.693196 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:15.693166 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-226f1f-84f9b576c9-kpxvc"] Jul 07 22:32:16.576596 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:16.576559 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbf3578-be4c-406a-af40-c091891241ec" path="/var/lib/kubelet/pods/5fbf3578-be4c-406a-af40-c091891241ec/volumes" Jul 07 22:32:20.390203 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:20.390162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:20.453103 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:20.453064 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:20.658324 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:20.658225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:21.186878 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:21.186847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:21.491512 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:21.491414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:21.723240 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:21.723199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:24.047519 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.047479 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:24.148868 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.148833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:24.293971 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.293933 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:24.394471 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.394364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:24.401923 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.401892 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx"] Jul 07 22:32:24.402392 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.402373 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985ba99d-7931-4860-97c2-e5fdc97c50bb" containerName="maas-api" Jul 07 22:32:24.402492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.402395 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="985ba99d-7931-4860-97c2-e5fdc97c50bb" containerName="maas-api" Jul 07 22:32:24.402492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.402409 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbf3578-be4c-406a-af40-c091891241ec" containerName="maas-api" Jul 07 22:32:24.402492 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.402417 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbf3578-be4c-406a-af40-c091891241ec" containerName="maas-api" Jul 07 22:32:24.402652 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.402532 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="985ba99d-7931-4860-97c2-e5fdc97c50bb" containerName="maas-api" Jul 07 22:32:24.402652 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.402544 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbf3578-be4c-406a-af40-c091891241ec" containerName="maas-api" Jul 07 22:32:24.405878 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.405854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.407879 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.407856 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7e3dc8-serving-cert\"" Jul 07 22:32:24.408035 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.407995 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w9p79\"" Jul 07 22:32:24.416226 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.416197 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx"] Jul 07 22:32:24.478172 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.478132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d620cd-6ab3-40be-a405-76c735a8e8f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.478410 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.478186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.478410 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.478306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvs5p\" (UniqueName: \"kubernetes.io/projected/52d620cd-6ab3-40be-a405-76c735a8e8f5-kube-api-access-rvs5p\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.496158 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.496126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:24.579751 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.579717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d620cd-6ab3-40be-a405-76c735a8e8f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.579751 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.579761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.579984 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.579813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvs5p\" (UniqueName: \"kubernetes.io/projected/52d620cd-6ab3-40be-a405-76c735a8e8f5-kube-api-access-rvs5p\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.579984 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:32:24.579933 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-serving-cert: secret "maas-api-e2e-shared-a-7e3dc8-serving-cert" not found Jul 07 22:32:24.580086 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:32:24.580011 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls podName:52d620cd-6ab3-40be-a405-76c735a8e8f5 nodeName:}" failed. No retries permitted until 2026-07-07 22:32:25.079988089 +0000 UTC m=+2001.071561294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls") pod "maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" (UID: "52d620cd-6ab3-40be-a405-76c735a8e8f5") : secret "maas-api-e2e-shared-a-7e3dc8-serving-cert" not found Jul 07 22:32:24.580434 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.580415 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d620cd-6ab3-40be-a405-76c735a8e8f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.588657 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.588625 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvs5p\" (UniqueName: \"kubernetes.io/projected/52d620cd-6ab3-40be-a405-76c735a8e8f5-kube-api-access-rvs5p\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:24.963606 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:24.963573 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:25.064170 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:25.064137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:25.085307 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:25.085266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:25.087989 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:25.087961 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:25.318587 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:25.318545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:25.469336 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:25.469305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx"] Jul 07 22:32:25.471212 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:32:25.471176 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d620cd_6ab3_40be_a405_76c735a8e8f5.slice/crio-38a2779977a21f14c64918fe708978a3fc0e2a31d448c7fa71d3bd66a2868485 WatchSource:0}: Error finding container 38a2779977a21f14c64918fe708978a3fc0e2a31d448c7fa71d3bd66a2868485: Status 404 returned error can't find the container with id 38a2779977a21f14c64918fe708978a3fc0e2a31d448c7fa71d3bd66a2868485 Jul 07 22:32:25.723066 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:25.722960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" event={"ID":"52d620cd-6ab3-40be-a405-76c735a8e8f5","Type":"ContainerStarted","Data":"38a2779977a21f14c64918fe708978a3fc0e2a31d448c7fa71d3bd66a2868485"} Jul 07 22:32:26.729821 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:26.729782 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" event={"ID":"52d620cd-6ab3-40be-a405-76c735a8e8f5","Type":"ContainerStarted","Data":"c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2"} Jul 07 22:32:26.730280 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:26.729854 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:26.745692 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:26.745636 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" podStartSLOduration=2.273855296 podStartE2EDuration="2.745616485s" totalImagesPullingTime="471.761189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:32:26.743591338 +0000 UTC m=+2002.735164566" watchObservedRunningTime="2026-07-07 22:32:26.745616485 +0000 UTC m=+2002.737189712" Jul 07 22:32:28.814813 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:28.814778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:29.148632 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:29.148539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:29.459705 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:29.459607 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:29.738138 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:29.738043 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:32.608094 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.608056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:32.671629 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.671598 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t"] Jul 07 22:32:32.675391 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.675368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.677425 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.677405 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5b00ce-serving-cert\"" Jul 07 22:32:32.685439 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.685404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t"] Jul 07 22:32:32.735737 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.735704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:32.739803 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.739779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:32:32.763755 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.763719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.763978 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.763859 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/4bf036f6-0536-483f-929b-ff9766916607-kube-api-access-w87tw\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.763978 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.763910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf036f6-0536-483f-929b-ff9766916607-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.864777 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.864664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/4bf036f6-0536-483f-929b-ff9766916607-kube-api-access-w87tw\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.864980 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.864847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf036f6-0536-483f-929b-ff9766916607-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.864980 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.864939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.865141 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:32:32.865120 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-serving-cert: secret "maas-api-e2e-shared-b-5b00ce-serving-cert" not found Jul 07 22:32:32.865201 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:32:32.865190 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls podName:4bf036f6-0536-483f-929b-ff9766916607 nodeName:}" failed. No retries permitted until 2026-07-07 22:32:33.365170289 +0000 UTC m=+2009.356743496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls") pod "maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" (UID: "4bf036f6-0536-483f-929b-ff9766916607") : secret "maas-api-e2e-shared-b-5b00ce-serving-cert" not found Jul 07 22:32:32.865617 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.865600 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf036f6-0536-483f-929b-ff9766916607-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:32.890977 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.890940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:32.895771 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:32.895744 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/4bf036f6-0536-483f-929b-ff9766916607-kube-api-access-w87tw\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:33.063877 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.063842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:33.371012 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.370973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:33.373632 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.373603 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls\") pod \"maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:33.489037 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.488979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:33.588070 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.588004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:33.589957 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.589918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:33.745365 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.745330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t"] Jul 07 22:32:33.750274 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:32:33.750245 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf036f6_0536_483f_929b_ff9766916607.slice/crio-1321a4c98af9b5943b06440b5902d7bdce086ec2611d76680ae3abdd34ad80a7 WatchSource:0}: Error finding container 1321a4c98af9b5943b06440b5902d7bdce086ec2611d76680ae3abdd34ad80a7: Status 404 returned error can't find the container with id 1321a4c98af9b5943b06440b5902d7bdce086ec2611d76680ae3abdd34ad80a7 Jul 07 22:32:33.765862 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:33.765824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" event={"ID":"4bf036f6-0536-483f-929b-ff9766916607","Type":"ContainerStarted","Data":"1321a4c98af9b5943b06440b5902d7bdce086ec2611d76680ae3abdd34ad80a7"} Jul 07 22:32:34.771961 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:34.771921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" event={"ID":"4bf036f6-0536-483f-929b-ff9766916607","Type":"ContainerStarted","Data":"ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199"} Jul 07 22:32:34.772423 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:34.772061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:34.789149 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:34.789090 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" podStartSLOduration=2.429447147 podStartE2EDuration="2.789072009s" totalImagesPullingTime="359.624862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:32:34.786767518 +0000 UTC m=+2010.778340747" watchObservedRunningTime="2026-07-07 22:32:34.789072009 +0000 UTC m=+2010.780645236" Jul 07 22:32:35.161273 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:35.161178 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:37.043610 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:37.043571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:40.783703 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:40.783675 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:32:46.547668 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:46.547637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:46.649269 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:46.649233 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:54.386175 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:54.386139 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:32:54.487138 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:32:54.487099 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:33:44.283647 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:33:44.283543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:33:44.384003 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:33:44.383961 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:15.884808 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:15.884716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:39.792424 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:39.792387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:41.968866 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:41.968829 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:42.398627 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.398560 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:42.499490 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.499437 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:42.618289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.618249 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx"] Jul 07 22:35:42.618562 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.618535 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" podUID="52d620cd-6ab3-40be-a405-76c735a8e8f5" containerName="maas-api" containerID="cri-o://c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2" gracePeriod=30 Jul 07 22:35:42.652706 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.652609 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:42.735163 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.735118 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" podUID="52d620cd-6ab3-40be-a405-76c735a8e8f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.52:8443/health\": dial tcp 10.134.0.52:8443: connect: connection refused" Jul 07 22:35:42.875249 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.875219 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:35:42.923164 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.923074 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls\") pod \"52d620cd-6ab3-40be-a405-76c735a8e8f5\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " Jul 07 22:35:42.923164 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.923141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvs5p\" (UniqueName: \"kubernetes.io/projected/52d620cd-6ab3-40be-a405-76c735a8e8f5-kube-api-access-rvs5p\") pod \"52d620cd-6ab3-40be-a405-76c735a8e8f5\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " Jul 07 22:35:42.923348 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.923197 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d620cd-6ab3-40be-a405-76c735a8e8f5-openshift-service-ca\") pod \"52d620cd-6ab3-40be-a405-76c735a8e8f5\" (UID: \"52d620cd-6ab3-40be-a405-76c735a8e8f5\") " Jul 07 22:35:42.923604 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.923580 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d620cd-6ab3-40be-a405-76c735a8e8f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52d620cd-6ab3-40be-a405-76c735a8e8f5" (UID: "52d620cd-6ab3-40be-a405-76c735a8e8f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:35:42.925389 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.925361 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d620cd-6ab3-40be-a405-76c735a8e8f5-kube-api-access-rvs5p" (OuterVolumeSpecName: "kube-api-access-rvs5p") pod "52d620cd-6ab3-40be-a405-76c735a8e8f5" (UID: "52d620cd-6ab3-40be-a405-76c735a8e8f5"). InnerVolumeSpecName "kube-api-access-rvs5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:35:42.925498 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:42.925390 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52d620cd-6ab3-40be-a405-76c735a8e8f5" (UID: "52d620cd-6ab3-40be-a405-76c735a8e8f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:35:43.024502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.024456 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52d620cd-6ab3-40be-a405-76c735a8e8f5-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:35:43.024502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.024493 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52d620cd-6ab3-40be-a405-76c735a8e8f5-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:35:43.024502 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.024502 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvs5p\" (UniqueName: \"kubernetes.io/projected/52d620cd-6ab3-40be-a405-76c735a8e8f5-kube-api-access-rvs5p\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:35:43.729135 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.729095 2565 generic.go:356] "Generic (PLEG): container finished" podID="52d620cd-6ab3-40be-a405-76c735a8e8f5" containerID="c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2" exitCode=0 Jul 07 22:35:43.729377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.729141 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" Jul 07 22:35:43.729377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.729182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" event={"ID":"52d620cd-6ab3-40be-a405-76c735a8e8f5","Type":"ContainerDied","Data":"c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2"} Jul 07 22:35:43.729377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.729242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx" event={"ID":"52d620cd-6ab3-40be-a405-76c735a8e8f5","Type":"ContainerDied","Data":"38a2779977a21f14c64918fe708978a3fc0e2a31d448c7fa71d3bd66a2868485"} Jul 07 22:35:43.729377 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.729271 2565 scope.go:117] "RemoveContainer" containerID="c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2" Jul 07 22:35:43.740337 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.740311 2565 scope.go:117] "RemoveContainer" containerID="c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2" Jul 07 22:35:43.740680 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:35:43.740651 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2\": container with ID starting with c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2 not found: ID does not exist" containerID="c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2" Jul 07 22:35:43.740779 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.740693 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2"} err="failed to get container status \"c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2\": rpc error: code = NotFound desc = could not find container \"c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2\": container with ID starting with c072dd64cfff3c8ee18b82d852f5d05295c7413a7ddcbb97adc25e84adacb5e2 not found: ID does not exist" Jul 07 22:35:43.749930 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.749894 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx"] Jul 07 22:35:43.753412 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:43.753382 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e3dc8-8f7786c86-nm9rx"] Jul 07 22:35:44.576652 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:44.576609 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d620cd-6ab3-40be-a405-76c735a8e8f5" path="/var/lib/kubelet/pods/52d620cd-6ab3-40be-a405-76c735a8e8f5/volumes" Jul 07 22:35:47.978617 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:47.978576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:48.079256 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:48.079219 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:48.238846 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:48.238753 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:48.645493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:48.645457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:49.037852 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.037818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:49.159970 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.159939 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:49.274258 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.274218 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:49.396172 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.396081 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t"] Jul 07 22:35:49.396349 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.396306 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" podUID="4bf036f6-0536-483f-929b-ff9766916607" containerName="maas-api" containerID="cri-o://ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199" gracePeriod=30 Jul 07 22:35:49.421407 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.421369 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:49.648150 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.648068 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:35:49.763090 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.763012 2565 generic.go:356] "Generic (PLEG): container finished" podID="4bf036f6-0536-483f-929b-ff9766916607" containerID="ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199" exitCode=0 Jul 07 22:35:49.763296 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.763099 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" Jul 07 22:35:49.763296 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.763101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" event={"ID":"4bf036f6-0536-483f-929b-ff9766916607","Type":"ContainerDied","Data":"ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199"} Jul 07 22:35:49.763296 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.763159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t" event={"ID":"4bf036f6-0536-483f-929b-ff9766916607","Type":"ContainerDied","Data":"1321a4c98af9b5943b06440b5902d7bdce086ec2611d76680ae3abdd34ad80a7"} Jul 07 22:35:49.763296 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.763188 2565 scope.go:117] "RemoveContainer" containerID="ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199" Jul 07 22:35:49.771859 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.771838 2565 scope.go:117] "RemoveContainer" containerID="ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199" Jul 07 22:35:49.772214 ip-10-0-130-137 kubenswrapper[2565]: E0707 22:35:49.772192 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199\": container with ID starting with ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199 not found: ID does not exist" containerID="ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199" Jul 07 22:35:49.772286 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.772226 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199"} err="failed to get container status \"ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199\": rpc error: code = NotFound desc = could not find container \"ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199\": container with ID starting with ed85fab33f51436f0f413373bd9f7cd50dfb3b6eb3d7d6589f61b874b10d5199 not found: ID does not exist" Jul 07 22:35:49.787586 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.787552 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls\") pod \"4bf036f6-0536-483f-929b-ff9766916607\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " Jul 07 22:35:49.787785 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.787654 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/4bf036f6-0536-483f-929b-ff9766916607-kube-api-access-w87tw\") pod \"4bf036f6-0536-483f-929b-ff9766916607\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " Jul 07 22:35:49.787785 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.787744 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf036f6-0536-483f-929b-ff9766916607-openshift-service-ca\") pod \"4bf036f6-0536-483f-929b-ff9766916607\" (UID: \"4bf036f6-0536-483f-929b-ff9766916607\") " Jul 07 22:35:49.788179 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.788151 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf036f6-0536-483f-929b-ff9766916607-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bf036f6-0536-483f-929b-ff9766916607" (UID: "4bf036f6-0536-483f-929b-ff9766916607"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 22:35:49.790018 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.789992 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bf036f6-0536-483f-929b-ff9766916607" (UID: "4bf036f6-0536-483f-929b-ff9766916607"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 22:35:49.790152 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.789997 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf036f6-0536-483f-929b-ff9766916607-kube-api-access-w87tw" (OuterVolumeSpecName: "kube-api-access-w87tw") pod "4bf036f6-0536-483f-929b-ff9766916607" (UID: "4bf036f6-0536-483f-929b-ff9766916607"). InnerVolumeSpecName "kube-api-access-w87tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 22:35:49.888674 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.888625 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/4bf036f6-0536-483f-929b-ff9766916607-kube-api-access-w87tw\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:35:49.888674 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.888664 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf036f6-0536-483f-929b-ff9766916607-openshift-service-ca\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:35:49.888674 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:49.888674 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf036f6-0536-483f-929b-ff9766916607-maas-api-tls\") on node \"ip-10-0-130-137.ec2.internal\" DevicePath \"\"" Jul 07 22:35:50.086533 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:50.086490 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t"] Jul 07 22:35:50.097017 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:50.096988 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5b00ce-58d8df5754-krq6t"] Jul 07 22:35:50.581092 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:50.581050 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf036f6-0536-483f-929b-ff9766916607" path="/var/lib/kubelet/pods/4bf036f6-0536-483f-929b-ff9766916607/volumes" Jul 07 22:35:54.696858 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:54.696823 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:35:55.835248 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:35:55.835215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:36:00.699817 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:00.699787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d68568969-cxrjg_94c39d98-31c9-4bb1-b998-ff1a9be627f1/manager/0.log" Jul 07 22:36:01.353433 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:01.353402 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd55458bc-kj6zw_ae6f99ef-f667-4660-8361-d028816dc07f/copy-manifests/0.log" Jul 07 22:36:01.359415 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:01.359388 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd55458bc-kj6zw_ae6f99ef-f667-4660-8361-d028816dc07f/manager/0.log" Jul 07 22:36:03.537827 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:03.537795 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6kpw7_005c655e-4c7f-4777-acb4-68e99c1d299e/manager/0.log" Jul 07 22:36:04.256607 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:04.256572 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98fbf9c8c-xh7bf_099075b9-b650-496c-88be-c9c278c030ab/kube-auth-proxy/0.log" Jul 07 22:36:05.327535 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:05.327509 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-26r48_8249493d-51c6-4300-a89d-1566194ad24a/main/0.log" Jul 07 22:36:09.547869 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.547830 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wh6z7/must-gather-7vxlb"] Jul 07 22:36:09.548459 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.548438 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d620cd-6ab3-40be-a405-76c735a8e8f5" containerName="maas-api" Jul 07 22:36:09.548540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.548462 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d620cd-6ab3-40be-a405-76c735a8e8f5" containerName="maas-api" Jul 07 22:36:09.548540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.548484 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf036f6-0536-483f-929b-ff9766916607" containerName="maas-api" Jul 07 22:36:09.548540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.548492 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf036f6-0536-483f-929b-ff9766916607" containerName="maas-api" Jul 07 22:36:09.548701 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.548598 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf036f6-0536-483f-929b-ff9766916607" containerName="maas-api" Jul 07 22:36:09.548701 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.548613 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d620cd-6ab3-40be-a405-76c735a8e8f5" containerName="maas-api" Jul 07 22:36:09.551923 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.551900 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.554219 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.554196 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wh6z7\"/\"kube-root-ca.crt\"" Jul 07 22:36:09.554669 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.554642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wh6z7\"/\"default-dockercfg-bhddq\"" Jul 07 22:36:09.554727 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.554642 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wh6z7\"/\"openshift-service-ca.crt\"" Jul 07 22:36:09.560170 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.560066 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:36:09.560733 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.560713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh6z7/must-gather-7vxlb"] Jul 07 22:36:09.619637 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.619602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" limit=1024 Jul 07 22:36:09.674462 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.674428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqxk\" (UniqueName: \"kubernetes.io/projected/4985a580-b59b-4c2f-8d71-f3a4c08542ab-kube-api-access-thqxk\") pod \"must-gather-7vxlb\" (UID: \"4985a580-b59b-4c2f-8d71-f3a4c08542ab\") " pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.674659 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.674586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4985a580-b59b-4c2f-8d71-f3a4c08542ab-must-gather-output\") pod \"must-gather-7vxlb\" (UID: \"4985a580-b59b-4c2f-8d71-f3a4c08542ab\") " pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.775515 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.775472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4985a580-b59b-4c2f-8d71-f3a4c08542ab-must-gather-output\") pod \"must-gather-7vxlb\" (UID: \"4985a580-b59b-4c2f-8d71-f3a4c08542ab\") " pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.775515 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.775517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqxk\" (UniqueName: \"kubernetes.io/projected/4985a580-b59b-4c2f-8d71-f3a4c08542ab-kube-api-access-thqxk\") pod \"must-gather-7vxlb\" (UID: \"4985a580-b59b-4c2f-8d71-f3a4c08542ab\") " pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.775884 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.775860 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4985a580-b59b-4c2f-8d71-f3a4c08542ab-must-gather-output\") pod \"must-gather-7vxlb\" (UID: \"4985a580-b59b-4c2f-8d71-f3a4c08542ab\") " pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.782936 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.782908 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqxk\" (UniqueName: \"kubernetes.io/projected/4985a580-b59b-4c2f-8d71-f3a4c08542ab-kube-api-access-thqxk\") pod \"must-gather-7vxlb\" (UID: \"4985a580-b59b-4c2f-8d71-f3a4c08542ab\") " pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:09.862386 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:09.862289 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh6z7/must-gather-7vxlb" Jul 07 22:36:10.021018 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:10.020954 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh6z7/must-gather-7vxlb"] Jul 07 22:36:10.023526 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:36:10.023496 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4985a580_b59b_4c2f_8d71_f3a4c08542ab.slice/crio-0df2c66af99b8dac99213e6c987f257421edf9576cd910d1480f6221358a480e WatchSource:0}: Error finding container 0df2c66af99b8dac99213e6c987f257421edf9576cd910d1480f6221358a480e: Status 404 returned error can't find the container with id 0df2c66af99b8dac99213e6c987f257421edf9576cd910d1480f6221358a480e Jul 07 22:36:10.025389 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:10.025371 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 22:36:10.890678 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:10.890636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh6z7/must-gather-7vxlb" event={"ID":"4985a580-b59b-4c2f-8d71-f3a4c08542ab","Type":"ContainerStarted","Data":"0df2c66af99b8dac99213e6c987f257421edf9576cd910d1480f6221358a480e"} Jul 07 22:36:11.905202 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:11.905155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh6z7/must-gather-7vxlb" event={"ID":"4985a580-b59b-4c2f-8d71-f3a4c08542ab","Type":"ContainerStarted","Data":"650889bc4b57fcc9dd07f4cfdd3c1e295de32762e4e0540845720ac8d4097ab8"} Jul 07 22:36:11.905812 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:11.905764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh6z7/must-gather-7vxlb" event={"ID":"4985a580-b59b-4c2f-8d71-f3a4c08542ab","Type":"ContainerStarted","Data":"4295ff0e8a5e6d10f3c48a81e7182ff919475d0bcdad463c9c6bbf0ef0b6567b"} Jul 07 22:36:11.921677 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:11.921615 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wh6z7/must-gather-7vxlb" podStartSLOduration=2.014222337 podStartE2EDuration="2.921589697s" totalImagesPullingTime="907.36736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 22:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:36:11.919872197 +0000 UTC m=+2227.911445430" watchObservedRunningTime="2026-07-07 22:36:11.921589697 +0000 UTC m=+2227.913162925" Jul 07 22:36:12.793282 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:12.793247 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cz4t2_f9b71ef0-3555-4870-9392-4955495398c0/global-pull-secret-syncer/0.log" Jul 07 22:36:12.921242 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:12.921211 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h8n52_8a41a985-5a2d-425b-8723-2fffc40a1754/konnectivity-agent/0.log" Jul 07 22:36:13.031836 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:13.031799 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-137.ec2.internal_b5456099913828cc9faa58b08483592f/haproxy/0.log" Jul 07 22:36:18.785855 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:18.785811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6kpw7_005c655e-4c7f-4777-acb4-68e99c1d299e/manager/0.log" Jul 07 22:36:20.379685 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.379644 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/alertmanager/0.log" Jul 07 22:36:20.403520 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.403462 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/config-reloader/0.log" Jul 07 22:36:20.433839 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.433742 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/kube-rbac-proxy-web/0.log" Jul 07 22:36:20.456262 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.456231 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/kube-rbac-proxy/0.log" Jul 07 22:36:20.488931 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.488904 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/kube-rbac-proxy-metric/0.log" Jul 07 22:36:20.517811 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.517728 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/prom-label-proxy/0.log" Jul 07 22:36:20.543733 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.543231 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_42ecc177-a45a-42f7-855f-17e2b6a13956/init-config-reloader/0.log" Jul 07 22:36:20.631461 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.630966 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-j28pk_2abe1af1-919a-44da-8ab8-41598348bd64/kube-state-metrics/0.log" Jul 07 22:36:20.660085 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.660014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-j28pk_2abe1af1-919a-44da-8ab8-41598348bd64/kube-rbac-proxy-main/0.log" Jul 07 22:36:20.688440 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.688342 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-j28pk_2abe1af1-919a-44da-8ab8-41598348bd64/kube-rbac-proxy-self/0.log" Jul 07 22:36:20.719344 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.719307 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78987745dc-qb8sq_d2bf5db6-ab88-4f7b-a3f9-2f012b420d50/metrics-server/0.log" Jul 07 22:36:20.747376 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.747259 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-rlb9n_08d529f5-c2d4-4a41-805b-20ba625634c7/monitoring-plugin/0.log" Jul 07 22:36:20.783200 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.783104 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cgs9n_02da7917-ab60-4340-bbdf-cb39faf23ecc/node-exporter/0.log" Jul 07 22:36:20.806493 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.806453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cgs9n_02da7917-ab60-4340-bbdf-cb39faf23ecc/kube-rbac-proxy/0.log" Jul 07 22:36:20.830007 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:20.829970 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cgs9n_02da7917-ab60-4340-bbdf-cb39faf23ecc/init-textfile/0.log" Jul 07 22:36:21.037731 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.037693 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-q5wfs_78c08197-161f-41a0-8ed1-397c0eafef03/kube-rbac-proxy-main/0.log" Jul 07 22:36:21.079503 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.079426 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-q5wfs_78c08197-161f-41a0-8ed1-397c0eafef03/kube-rbac-proxy-self/0.log" Jul 07 22:36:21.098384 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.098340 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-q5wfs_78c08197-161f-41a0-8ed1-397c0eafef03/openshift-state-metrics/0.log" Jul 07 22:36:21.139222 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.138980 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/prometheus/0.log" Jul 07 22:36:21.161942 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.161905 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/config-reloader/0.log" Jul 07 22:36:21.184236 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.184179 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/thanos-sidecar/0.log" Jul 07 22:36:21.205261 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.205215 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/kube-rbac-proxy-web/0.log" Jul 07 22:36:21.226713 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.226678 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/kube-rbac-proxy/0.log" Jul 07 22:36:21.253192 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.253161 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/kube-rbac-proxy-thanos/0.log" Jul 07 22:36:21.275102 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.275001 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_52d2d8a8-c7fe-4ffb-89f0-72732478793a/init-config-reloader/0.log" Jul 07 22:36:21.304274 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.304236 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-8psz8_e1ae39a6-fee6-45b5-93fb-9b6a82727a68/prometheus-operator/0.log" Jul 07 22:36:21.325471 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.325435 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-8psz8_e1ae39a6-fee6-45b5-93fb-9b6a82727a68/kube-rbac-proxy/0.log" Jul 07 22:36:21.350530 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.350498 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-r5bww_66b7f667-f0c4-4919-8ebc-95fd8e6b1015/prometheus-operator-admission-webhook/0.log" Jul 07 22:36:21.384214 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.384182 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dcf7b559b-ppb5m_8b3bc5bf-dc00-44d6-aab2-8af876b3a17d/telemeter-client/0.log" Jul 07 22:36:21.403744 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.403713 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dcf7b559b-ppb5m_8b3bc5bf-dc00-44d6-aab2-8af876b3a17d/reload/0.log" Jul 07 22:36:21.425275 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.425241 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dcf7b559b-ppb5m_8b3bc5bf-dc00-44d6-aab2-8af876b3a17d/kube-rbac-proxy/0.log" Jul 07 22:36:21.457095 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.457067 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f5c557b9-kgh6m_3d2bc457-068c-407e-9864-92cbcd49e838/thanos-query/0.log" Jul 07 22:36:21.485491 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.485463 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f5c557b9-kgh6m_3d2bc457-068c-407e-9864-92cbcd49e838/kube-rbac-proxy-web/0.log" Jul 07 22:36:21.507189 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.507159 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f5c557b9-kgh6m_3d2bc457-068c-407e-9864-92cbcd49e838/kube-rbac-proxy/0.log" Jul 07 22:36:21.528197 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.528101 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f5c557b9-kgh6m_3d2bc457-068c-407e-9864-92cbcd49e838/prom-label-proxy/0.log" Jul 07 22:36:21.562379 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.562337 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f5c557b9-kgh6m_3d2bc457-068c-407e-9864-92cbcd49e838/kube-rbac-proxy-rules/0.log" Jul 07 22:36:21.584251 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:21.584221 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f5c557b9-kgh6m_3d2bc457-068c-407e-9864-92cbcd49e838/kube-rbac-proxy-metrics/0.log" Jul 07 22:36:23.322881 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.322836 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p"] Jul 07 22:36:23.327276 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.327246 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.338609 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.338303 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p"] Jul 07 22:36:23.431880 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.431504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-lib-modules\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.431880 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.431624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-proc\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.431880 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.431653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqng\" (UniqueName: \"kubernetes.io/projected/59c374ae-1611-41d3-9990-6325e2307688-kube-api-access-twqng\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.431880 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.431709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-sys\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.431880 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.431743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-podres\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532796 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-proc\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqng\" (UniqueName: \"kubernetes.io/projected/59c374ae-1611-41d3-9990-6325e2307688-kube-api-access-twqng\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-sys\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-podres\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532898 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-proc\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-lib-modules\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.532992 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.532972 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-sys\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.533309 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.533039 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-podres\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.533309 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.533100 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59c374ae-1611-41d3-9990-6325e2307688-lib-modules\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.541290 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.541256 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqng\" (UniqueName: \"kubernetes.io/projected/59c374ae-1611-41d3-9990-6325e2307688-kube-api-access-twqng\") pod \"perf-node-gather-daemonset-r288p\" (UID: \"59c374ae-1611-41d3-9990-6325e2307688\") " pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.644383 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.644268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:23.847684 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:23.847509 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p"] Jul 07 22:36:23.855397 ip-10-0-130-137 kubenswrapper[2565]: W0707 22:36:23.853730 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod59c374ae_1611_41d3_9990_6325e2307688.slice/crio-dd111ae79ca711583e9b34471eb1534f1723042d19fb33a4d6be8c39e4751336 WatchSource:0}: Error finding container dd111ae79ca711583e9b34471eb1534f1723042d19fb33a4d6be8c39e4751336: Status 404 returned error can't find the container with id dd111ae79ca711583e9b34471eb1534f1723042d19fb33a4d6be8c39e4751336 Jul 07 22:36:24.005059 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:24.004749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" event={"ID":"59c374ae-1611-41d3-9990-6325e2307688","Type":"ContainerStarted","Data":"e93a2830c1bdbf4a56e7c35bf7d5e5170e15079c1184bc123a06cc463e39accf"} Jul 07 22:36:24.005059 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:24.004817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" event={"ID":"59c374ae-1611-41d3-9990-6325e2307688","Type":"ContainerStarted","Data":"dd111ae79ca711583e9b34471eb1534f1723042d19fb33a4d6be8c39e4751336"} Jul 07 22:36:24.005330 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:24.005097 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:24.020924 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:24.020849 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" podStartSLOduration=1.020810629 podStartE2EDuration="1.020810629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 22:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 22:36:24.019327943 +0000 UTC m=+2240.010901172" watchObservedRunningTime="2026-07-07 22:36:24.020810629 +0000 UTC m=+2240.012383923" Jul 07 22:36:24.122754 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:24.122727 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-779fd78f6f-txpr5_6e008ede-34a9-462e-88d1-41575e11f1a5/console/0.log" Jul 07 22:36:25.815771 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:25.815692 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46cgj_62047d1c-5895-4bda-b8a0-3a57fbc5eeb9/dns/0.log" Jul 07 22:36:25.835289 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:25.835257 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46cgj_62047d1c-5895-4bda-b8a0-3a57fbc5eeb9/kube-rbac-proxy/0.log" Jul 07 22:36:25.952628 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:25.952599 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7k7p6_6ede2f80-924e-4425-92f0-6bc4ac661c93/dns-node-resolver/0.log" Jul 07 22:36:26.554264 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:26.554234 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8m9l6_48345d08-5995-4c23-a63c-c55b32e8d7cd/node-ca/0.log" Jul 07 22:36:27.696206 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:27.696178 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98fbf9c8c-xh7bf_099075b9-b650-496c-88be-c9c278c030ab/kube-auth-proxy/0.log" Jul 07 22:36:28.567462 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:28.567423 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4s4fw_71c673e5-2c40-47bb-92d5-0d73da843886/serve-healthcheck-canary/0.log" Jul 07 22:36:29.115454 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:29.115372 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9q6n_8673fd68-bfb5-4761-950b-bdc1d1ac5020/kube-rbac-proxy/0.log" Jul 07 22:36:29.138540 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:29.138508 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9q6n_8673fd68-bfb5-4761-950b-bdc1d1ac5020/exporter/0.log" Jul 07 22:36:29.159805 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:29.159767 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9q6n_8673fd68-bfb5-4761-950b-bdc1d1ac5020/extractor/0.log" Jul 07 22:36:30.023950 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:30.023908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wh6z7/perf-node-gather-daemonset-r288p" Jul 07 22:36:31.667531 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:31.667477 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d68568969-cxrjg_94c39d98-31c9-4bb1-b998-ff1a9be627f1/manager/0.log" Jul 07 22:36:31.921743 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:31.921660 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd55458bc-kj6zw_ae6f99ef-f667-4660-8361-d028816dc07f/manager/0.log" Jul 07 22:36:31.943451 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:31.943421 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dd55458bc-kj6zw_ae6f99ef-f667-4660-8361-d028816dc07f/copy-manifests/0.log" Jul 07 22:36:33.370544 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:33.370506 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c4f8f67f4-l8gx5_913c53c1-f6bb-46c1-a02a-998e51577862/manager/0.log" Jul 07 22:36:34.721950 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:34.721922 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98fbf9c8c-xh7bf_099075b9-b650-496c-88be-c9c278c030ab/kube-auth-proxy/0.log" Jul 07 22:36:36.373715 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:36.373646 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98fbf9c8c-xh7bf_099075b9-b650-496c-88be-c9c278c030ab/kube-auth-proxy/0.log" Jul 07 22:36:40.303697 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.303667 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/kube-multus-additional-cni-plugins/0.log" Jul 07 22:36:40.325758 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.325727 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/egress-router-binary-copy/0.log" Jul 07 22:36:40.345682 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.345657 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/cni-plugins/0.log" Jul 07 22:36:40.364812 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.364777 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/bond-cni-plugin/0.log" Jul 07 22:36:40.383920 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.383892 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/routeoverride-cni/0.log" Jul 07 22:36:40.403488 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.403451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/whereabouts-cni-bincopy/0.log" Jul 07 22:36:40.422862 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.422827 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnfnp_c0b7e5bc-a55f-4475-a9ce-a3f386594731/whereabouts-cni/0.log" Jul 07 22:36:40.635604 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.635577 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w8gkx_402eead4-ad2d-4876-9e68-54be18236c77/kube-multus/0.log" Jul 07 22:36:40.657694 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.657665 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4zzjj_17749b82-0e66-4e1d-a410-a6f414f06ff1/network-metrics-daemon/0.log" Jul 07 22:36:40.675192 ip-10-0-130-137 kubenswrapper[2565]: I0707 22:36:40.675163 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4zzjj_17749b82-0e66-4e1d-a410-a6f414f06ff1/kube-rbac-proxy/0.log"