Jul 21 20:58:36.352584 ip-10-0-132-65 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 20:58:36.352597 ip-10-0-132-65 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 20:58:36.352604 ip-10-0-132-65 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 20:58:36.352897 ip-10-0-132-65 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 20:58:46.551435 ip-10-0-132-65 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 20:58:46.551458 ip-10-0-132-65 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7080bf1490af4b4aa6427d80d7c7ff65 -- Jul 21 21:00:22.942630 ip-10-0-132-65 systemd[1]: Starting Kubernetes Kubelet... Jul 21 21:00:23.350147 ip-10-0-132-65 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:00:23.350147 ip-10-0-132-65 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 21:00:23.350147 ip-10-0-132-65 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:00:23.350147 ip-10-0-132-65 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 21:00:23.350147 ip-10-0-132-65 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:00:23.351086 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.351009 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 21:00:23.355429 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355415 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:23.355429 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355429 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355432 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355435 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355438 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355441 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355444 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355447 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355449 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355452 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355454 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355458 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355461 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355463 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355473 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355476 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355478 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355481 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355483 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355486 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355488 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:23.355501 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355491 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355493 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355496 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355498 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355501 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355503 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355506 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355509 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355511 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355514 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355517 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355520 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355523 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355525 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355528 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355531 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355533 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355537 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355539 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355541 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:23.355979 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355544 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355546 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355548 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355551 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355553 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355555 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355558 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355560 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355562 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355565 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355567 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355569 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355580 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355583 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355586 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355589 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355592 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355595 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355597 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355600 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:23.356498 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355602 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355607 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355611 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355614 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355617 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355620 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355624 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355628 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355632 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355635 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355639 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355641 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355644 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355647 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355650 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355652 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355655 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355658 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355660 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:23.357008 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355663 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355665 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355668 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355671 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355673 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355676 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355679 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355681 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355683 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355686 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355690 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355693 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355696 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355699 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355703 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355705 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355708 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355710 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:23.357494 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.355713 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:23.357917 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357903 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:23.357917 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357911 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:23.357917 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357916 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357922 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357927 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357931 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357934 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357937 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357940 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357942 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357945 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357948 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357950 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357953 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357955 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357957 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357960 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357962 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357964 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357967 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357969 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357972 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:23.357994 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357975 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357978 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357980 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357983 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357985 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357988 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357990 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357992 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357995 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.357998 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358000 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358003 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358006 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358008 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358011 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358013 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358016 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358018 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358021 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358023 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:23.358505 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358025 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358028 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358030 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358032 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358034 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358037 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358039 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358041 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358044 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358046 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358048 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358051 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358054 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358056 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358060 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358062 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358065 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358068 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358070 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358072 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:23.359016 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358075 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358077 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358080 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358082 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358084 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358086 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358089 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358091 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358095 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358099 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358102 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358104 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358107 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358110 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358112 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358115 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358119 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358122 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358125 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:23.359519 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358128 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358131 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358134 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358138 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358141 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358143 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358146 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358150 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358153 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358155 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358158 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358161 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358165 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358168 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358171 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358174 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358176 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358179 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358542 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358548 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:23.360090 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358551 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358554 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358559 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358562 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358565 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358568 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358571 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358574 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358577 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358580 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358582 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358585 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358587 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358589 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358592 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358594 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358597 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358599 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358602 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:23.360610 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358605 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358608 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358611 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358613 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358615 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358618 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358621 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358623 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358625 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358628 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358630 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358633 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358635 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358638 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358641 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358644 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358648 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358651 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358654 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358656 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:23.361059 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358659 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358661 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358664 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358666 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358669 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358671 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358673 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358676 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358678 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358681 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358683 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358685 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358689 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358691 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358694 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358696 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358700 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358702 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358705 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358707 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:23.361590 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358710 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358712 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358714 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358717 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358719 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358722 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358724 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358727 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358729 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358732 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358735 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358737 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358739 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358742 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358744 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358747 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358749 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358751 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358754 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:23.362061 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358756 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358758 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358761 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358763 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358766 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358769 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358771 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358774 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358776 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358779 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358781 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358784 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358786 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358789 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358791 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358794 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358797 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358800 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.358803 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358875 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 21:00:23.362527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358881 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358886 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358890 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358895 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358898 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358901 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358905 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358908 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358911 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358914 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358917 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358920 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358923 2570 flags.go:64] FLAG: --cgroup-root="" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358925 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358928 2570 flags.go:64] FLAG: --client-ca-file="" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358933 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358936 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358939 2570 flags.go:64] FLAG: --cluster-domain="" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358942 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358945 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358948 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358951 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358954 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 21:00:23.363032 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358957 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358959 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358962 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358965 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358968 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358971 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358973 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358976 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358980 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358983 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358985 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358988 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358990 2570 flags.go:64] FLAG: --enable-server="true" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358994 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.358998 2570 flags.go:64] FLAG: --event-burst="100" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359001 2570 flags.go:64] FLAG: --event-qps="50" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359003 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359006 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359009 2570 flags.go:64] FLAG: --eviction-hard="" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359012 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359015 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359018 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359020 2570 flags.go:64] FLAG: --eviction-soft="" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359023 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359028 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 21:00:23.363603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359031 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359035 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359037 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359040 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359043 2570 flags.go:64] FLAG: --feature-gates="" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359046 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359049 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359052 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359055 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359058 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359060 2570 flags.go:64] FLAG: --help="false" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359063 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359066 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359069 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359072 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359075 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359078 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359081 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359083 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359086 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359089 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359093 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359096 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359099 2570 flags.go:64] FLAG: --kube-reserved="" Jul 21 21:00:23.364220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359101 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359104 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359107 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359109 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359112 2570 flags.go:64] FLAG: --lock-file="" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359115 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359117 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359121 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359130 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359134 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359137 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359139 2570 flags.go:64] FLAG: --logging-format="text" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359143 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359146 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359148 2570 flags.go:64] FLAG: --manifest-url="" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359151 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359155 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359157 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359161 2570 flags.go:64] FLAG: --max-pods="110" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359164 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359167 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359170 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359172 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359175 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359178 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 21:00:23.364823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359181 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359187 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359190 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359193 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359195 2570 flags.go:64] FLAG: --pod-cidr="" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359199 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359204 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359206 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359209 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359212 2570 flags.go:64] FLAG: --port="10250" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359215 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359217 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060d80ea3862e1551" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359220 2570 flags.go:64] FLAG: --qos-reserved="" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359223 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359227 2570 flags.go:64] FLAG: --register-node="true" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359229 2570 flags.go:64] FLAG: --register-with-taints="" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359233 2570 flags.go:64] FLAG: --registry-burst="10" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359237 2570 flags.go:64] FLAG: --registry-qps="5" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359240 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359242 2570 flags.go:64] FLAG: --reserved-memory="" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359246 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359248 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359252 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359254 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359257 2570 flags.go:64] FLAG: --runonce="false" Jul 21 21:00:23.365481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359260 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359262 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359265 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359268 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359271 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359274 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359276 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359279 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359282 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359285 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359287 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359290 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359293 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359296 2570 flags.go:64] FLAG: --system-cgroups="" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359298 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359304 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359306 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359309 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359312 2570 flags.go:64] FLAG: --tls-min-version="" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359315 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359317 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359321 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359324 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359327 2570 flags.go:64] FLAG: --v="2" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359332 2570 flags.go:64] FLAG: --version="false" Jul 21 21:00:23.366128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359335 2570 flags.go:64] FLAG: --vmodule="" Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359339 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.359342 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359461 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359465 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359468 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359471 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359474 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359477 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359479 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359482 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359484 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359487 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359490 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359492 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359495 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359497 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359500 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359502 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359504 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:23.366762 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359507 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359510 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359512 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359515 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359517 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359520 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359522 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359525 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359530 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359532 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359535 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359538 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359541 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359543 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359546 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359548 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359551 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359553 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359557 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:23.367245 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359560 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359562 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359565 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359567 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359570 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359579 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359583 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359586 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359588 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359591 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359593 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359596 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359598 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359600 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359603 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359608 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359611 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359614 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359617 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:23.367729 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359619 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359622 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359626 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359629 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359632 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359635 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359638 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359641 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359644 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359646 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359649 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359651 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359653 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359656 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359658 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359661 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359663 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359665 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359668 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359670 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:23.368188 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359673 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359675 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359677 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359680 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359682 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359685 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359687 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359689 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359692 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359696 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359699 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359701 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359703 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359706 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359711 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359714 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359717 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359721 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359723 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:23.368702 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359726 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359728 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359731 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359733 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.359735 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.360726 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.367557 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.367570 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.368383 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.368391 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368442 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368447 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368450 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368453 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:23.369146 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368456 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368459 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368461 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368464 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368466 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368469 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368471 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368474 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368476 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368478 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368481 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368484 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368486 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368490 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368492 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368496 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368500 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368503 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368507 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:23.369515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368511 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368514 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368517 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368520 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368522 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368525 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368528 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368531 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368533 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368536 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368538 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368541 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368543 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368546 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368548 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368551 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368553 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368556 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368558 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368560 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:23.369971 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368563 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368565 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368568 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368570 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368573 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368575 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368592 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368596 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368599 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368602 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368606 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368609 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368612 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368615 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368618 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368621 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368624 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368626 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368629 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368632 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:23.371041 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368635 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368638 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368641 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368643 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368646 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368649 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368651 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368654 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368657 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368659 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368661 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368664 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368666 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368669 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368671 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368674 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368676 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368678 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368681 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368683 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:23.371581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368685 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368688 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368690 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368692 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368694 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368697 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368699 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368702 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368704 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368707 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368710 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368712 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368715 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368718 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368720 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:23.372075 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368722 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.368727 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368833 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368837 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368839 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368842 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368845 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368847 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368850 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368853 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368855 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368857 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368860 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368862 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368864 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:23.372587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368867 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368869 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368871 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368874 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368876 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368879 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368881 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368883 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368886 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368888 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368890 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368892 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368895 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368897 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368900 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368905 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368907 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368910 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368912 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368915 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:23.372947 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368918 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368920 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368922 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368925 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368927 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368930 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368932 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368934 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368937 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368939 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368941 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368943 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368946 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368948 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368950 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368953 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368955 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368957 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368959 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:23.373442 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368963 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368967 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368969 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368972 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368975 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368978 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368980 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368982 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368985 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368987 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368990 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368992 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368994 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368997 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.368999 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369002 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369006 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369008 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369011 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369013 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:23.373891 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369016 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369018 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369021 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369023 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369025 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369028 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369030 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369033 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369035 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369037 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369040 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369042 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369044 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369047 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369049 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369051 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369055 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369059 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:23.374463 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369062 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369066 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369070 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369073 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369076 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369079 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369081 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369084 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:23.369086 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.369091 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.369180 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.371147 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.372396 2570 server.go:1019] "Starting client certificate rotation" Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.372504 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:00:23.375183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.374193 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:00:23.378062 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.378047 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:00:23.381695 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.381677 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:00:23.395866 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.395850 2570 log.go:25] "Validated CRI v1 runtime API" Jul 21 21:00:23.397816 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.397804 2570 log.go:25] "Validated CRI v1 image API" Jul 21 21:00:23.398567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.398549 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 21:00:23.401591 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.401559 2570 fs.go:135] Filesystem UUIDs: map[5d5a2327-b08a-4400-9e16-0b5741fcc465:/dev/nvme0n1p4 612f1a3c-13e8-4e64-b5fb-c53b5c304aa5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 21 21:00:23.401657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.401594 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 21:00:23.404321 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.404305 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:00:23.407061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.406963 2570 manager.go:216] Machine: {Timestamp:2026-07-21 21:00:23.406501397 +0000 UTC m=+0.351122454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100575 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eff058b6ccef49b30b63b21331047 SystemUUID:ec2eff05-8b6c-cef4-9b30-b63b21331047 BootID:7080bf14-90af-4b4a-a642-7d80d7c7ff65 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:ec:d8:11:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:ec:d8:11:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:1c:f5:00:35:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 21:00:23.407061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.407057 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 21:00:23.407170 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.407155 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 21:00:23.410054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.410032 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 21:00:23.410183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.410056 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 21:00:23.411137 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.411127 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 21:00:23.411174 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.411140 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 21:00:23.411174 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.411147 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:00:23.411924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.411913 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:00:23.413086 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.413076 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 21:00:23.415931 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.415920 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:00:23.416038 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416029 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 21:00:23.416071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416063 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 21 21:00:23.416098 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416080 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 21:00:23.416098 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416094 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 21:00:23.416150 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416109 2570 kubelet.go:398] "Adding apiserver pod source" Jul 21 21:00:23.416150 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416121 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 21:00:23.416652 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416636 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:00:23.416696 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.416665 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:00:23.419776 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.419754 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 21:00:23.421474 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.421457 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 21:00:23.421539 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.421494 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 21:00:23.422561 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422549 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 21:00:23.422596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422567 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 21:00:23.422596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422573 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 21:00:23.422596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422578 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 21:00:23.422596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422587 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 21:00:23.422596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422592 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 21:00:23.422723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422600 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 21:00:23.422723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422605 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 21:00:23.422723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422611 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 21:00:23.422723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422616 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 21:00:23.422723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422624 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 21:00:23.422723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.422628 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 21:00:23.423486 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.423476 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 21:00:23.423521 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.423486 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 21:00:23.425201 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.425185 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbpp8" Jul 21 21:00:23.426522 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.426509 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 21:00:23.426837 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.426821 2570 server.go:1268] "Started kubelet" Jul 21 21:00:23.427017 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.426985 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 21:00:23.427111 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.427031 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 21:00:23.427167 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.426992 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 21:00:23.427167 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.427040 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 21:00:23.427167 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.426994 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 21:00:23.427613 ip-10-0-132-65 systemd[1]: Started Kubernetes Kubelet. Jul 21 21:00:23.428677 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.428665 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 21 21:00:23.428973 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.428955 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 21:00:23.432106 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.432087 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 21:00:23.433265 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.433249 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbpp8" Jul 21 21:00:23.434171 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.434156 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 21:00:23.434237 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.434161 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 21:00:23.434534 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.434427 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.434964 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.434946 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 21:00:23.434964 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.434948 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 21:00:23.435103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.434980 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 21:00:23.435103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435063 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 21 21:00:23.435103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435066 2570 factory.go:55] Registering systemd factory Jul 21 21:00:23.435103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435084 2570 factory.go:223] Registration of the systemd container factory successfully Jul 21 21:00:23.435286 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435072 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 21 21:00:23.435354 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435341 2570 factory.go:153] Registering CRI-O factory Jul 21 21:00:23.435423 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435380 2570 factory.go:223] Registration of the crio container factory successfully Jul 21 21:00:23.435888 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435871 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 21:00:23.435944 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435906 2570 factory.go:103] Registering Raw factory Jul 21 21:00:23.435944 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.435923 2570 manager.go:1196] Started watching for new ooms in manager Jul 21 21:00:23.436260 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.436248 2570 manager.go:349] Starting recovery of all containers Jul 21 21:00:23.437013 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.435284 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-65.ec2.internal.18c469bf787eabce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-65.ec2.internal,UID:ip-10-0-132-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-65.ec2.internal,},FirstTimestamp:2026-07-21 21:00:23.426796494 +0000 UTC m=+0.371417550,LastTimestamp:2026-07-21 21:00:23.426796494 +0000 UTC m=+0.371417550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-65.ec2.internal,}" Jul 21 21:00:23.439110 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.439069 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 21:00:23.439212 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.439120 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 21:00:23.449199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.449182 2570 manager.go:354] Recovery completed Jul 21 21:00:23.453332 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.453321 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.461673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.461657 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.461732 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.461689 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.461732 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.461703 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.462188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462175 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 21:00:23.462188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462188 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 21:00:23.462307 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462205 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:00:23.462307 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462223 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.462917 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462896 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.462961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462926 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.462961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.462937 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.463460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.463446 2570 policy_none.go:49] "None policy: Start" Jul 21 21:00:23.463572 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.463466 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 21:00:23.463572 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.463479 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 21:00:23.464489 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.464476 2570 policy_none.go:47] "Start" Jul 21 21:00:23.510027 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.509885 2570 manager.go:337] "Starting Device Plugin manager" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.510049 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.510063 2570 server.go:87] "Starting device plugin registration server" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.510291 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.510305 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.510416 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.510486 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.510492 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.510943 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 21:00:23.511661 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.510994 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.548707 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.548682 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 21:00:23.549943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.549928 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 21:00:23.549991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.549964 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 21:00:23.549991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.549985 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 21:00:23.550071 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.550016 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 21:00:23.552225 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.552208 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:23.610907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.610854 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.611621 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.611608 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.611674 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.611637 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.611674 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.611652 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.611733 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.611679 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.619795 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.619783 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.619840 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.619802 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-65.ec2.internal\": node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.639944 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.639926 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.650189 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.650158 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal"] Jul 21 21:00:23.650236 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.650227 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.650939 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.650927 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.651006 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.650951 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.651006 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.650965 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.652392 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.652378 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.652534 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.652521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.652579 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.652549 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.653033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.653019 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.653099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.653043 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.653099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.653071 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.653099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.653083 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.653240 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.653107 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.653240 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.653116 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.654440 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.654424 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.654526 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.654450 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:23.655143 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.655129 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:23.655223 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.655151 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:23.655223 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.655176 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:23.678909 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.678889 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-65.ec2.internal\" not found" node="ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.683111 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.683097 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-65.ec2.internal\" not found" node="ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.736646 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.736626 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/771a9cee6459867329caacdbd2b313c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"771a9cee6459867329caacdbd2b313c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.736734 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.736649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bb4be778fdc4d1b1c65a09ca6ed53e9-config\") pod \"kube-apiserver-proxy-ip-10-0-132-65.ec2.internal\" (UID: \"5bb4be778fdc4d1b1c65a09ca6ed53e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.736734 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.736667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/771a9cee6459867329caacdbd2b313c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"771a9cee6459867329caacdbd2b313c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.740728 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.740710 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.837787 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.837759 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/771a9cee6459867329caacdbd2b313c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"771a9cee6459867329caacdbd2b313c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.837787 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.837724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/771a9cee6459867329caacdbd2b313c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"771a9cee6459867329caacdbd2b313c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.837966 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.837825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/771a9cee6459867329caacdbd2b313c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"771a9cee6459867329caacdbd2b313c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.837966 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.837845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bb4be778fdc4d1b1c65a09ca6ed53e9-config\") pod \"kube-apiserver-proxy-ip-10-0-132-65.ec2.internal\" (UID: \"5bb4be778fdc4d1b1c65a09ca6ed53e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.837966 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.837896 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bb4be778fdc4d1b1c65a09ca6ed53e9-config\") pod \"kube-apiserver-proxy-ip-10-0-132-65.ec2.internal\" (UID: \"5bb4be778fdc4d1b1c65a09ca6ed53e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.837966 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.837923 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/771a9cee6459867329caacdbd2b313c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"771a9cee6459867329caacdbd2b313c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.840785 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.840765 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.941624 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:23.941566 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:23.982739 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.982708 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:23.986266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:23.986251 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 21 21:00:24.042227 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.042197 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.142738 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.142709 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.243346 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.243275 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.343889 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.343860 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.372028 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.372005 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 21:00:24.372516 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.372128 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:00:24.385721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.385702 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:24.438408 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.438376 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 20:55:24 +0000 UTC" deadline="2028-04-13 15:05:34.612783116 +0000 UTC" Jul 21 21:00:24.438408 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.438402 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15162h5m10.174384021s" Jul 21 21:00:24.444284 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.444268 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.538589 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.538528 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:24.544892 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.544875 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.592310 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:24.592285 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771a9cee6459867329caacdbd2b313c6.slice/crio-1267c2d554af8ad3c8be2d19e02731a32b2e6ac7dc875aa1d1d030249aae4536 WatchSource:0}: Error finding container 1267c2d554af8ad3c8be2d19e02731a32b2e6ac7dc875aa1d1d030249aae4536: Status 404 returned error can't find the container with id 1267c2d554af8ad3c8be2d19e02731a32b2e6ac7dc875aa1d1d030249aae4536 Jul 21 21:00:24.596001 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.595986 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:00:24.643839 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:24.643818 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb4be778fdc4d1b1c65a09ca6ed53e9.slice/crio-1cefe474dd47d7a61e7653bff060a39d617aa807e67356aaa05ee1cb52c898b4 WatchSource:0}: Error finding container 1cefe474dd47d7a61e7653bff060a39d617aa807e67356aaa05ee1cb52c898b4: Status 404 returned error can't find the container with id 1cefe474dd47d7a61e7653bff060a39d617aa807e67356aaa05ee1cb52c898b4 Jul 21 21:00:24.644923 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.644910 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.745579 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:24.745552 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 21 21:00:24.835544 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.835481 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:24.935303 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.935272 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 21 21:00:24.953417 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.953399 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:00:24.954888 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.954874 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 21 21:00:24.966820 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:24.966802 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:00:25.417852 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.417816 2570 apiserver.go:52] "Watching apiserver" Jul 21 21:00:25.427128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.426953 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 21:00:25.427858 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.427464 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6","openshift-dns/node-resolver-wk9cv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal","openshift-multus/multus-additional-cni-plugins-jwt58","openshift-ovn-kubernetes/ovnkube-node-hph6g","kube-system/global-pull-secret-syncer-6ddkj","kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jq5pf","openshift-image-registry/node-ca-7cgd8","openshift-multus/multus-x2hj2","openshift-multus/network-metrics-daemon-twf9r","openshift-network-diagnostics/network-check-target-lpdk5","openshift-network-operator/iptables-alerter-tlkbc","kube-system/konnectivity-agent-8wghw"] Jul 21 21:00:25.430234 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.430215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.432829 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.432808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.432991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.432973 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.433389 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.433352 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 21:00:25.433643 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.433629 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zwpcz\"" Jul 21 21:00:25.433721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.433672 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 21:00:25.433781 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.433630 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.435600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.434878 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 21:00:25.435600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.435446 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.435600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.435509 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 21:00:25.435811 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.435724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.435866 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.435833 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.436562 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.436385 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5tpsj\"" Jul 21 21:00:25.437716 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.437232 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.438333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.438127 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zlzzs\"" Jul 21 21:00:25.438333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.438315 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 21:00:25.438611 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.438596 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 21:00:25.440067 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.439399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.440067 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.439665 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 21:00:25.440067 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.439816 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 21:00:25.440067 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.439939 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dcsnq\"" Jul 21 21:00:25.442239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.441599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.442239 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.441687 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:25.442239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.441761 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 21:00:25.442239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.442045 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9h4q\"" Jul 21 21:00:25.443034 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.442592 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 21:00:25.443034 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.442739 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.443034 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.442913 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 21:00:25.445185 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.443899 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 21:00:25.445185 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.444093 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.445185 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.444619 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.446311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-system-cni-dir\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446386 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446386 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446465 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ec019a3-7329-4860-a941-03bc6091f53f-agent-certs\") pod \"konnectivity-agent-8wghw\" (UID: \"5ec019a3-7329-4860-a941-03bc6091f53f\") " pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.446465 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-cni-bin\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446465 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-conf-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446558 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446474 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4dr\" (UniqueName: \"kubernetes.io/projected/423f86d5-93d3-4bc3-8f97-4a52730ebfef-kube-api-access-vq4dr\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446558 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446498 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c8e7275-b6ac-4522-99e3-b35330ebe02a-iptables-alerter-script\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.446558 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcmk\" (UniqueName: \"kubernetes.io/projected/3f1ce5f0-5af3-45e3-9751-43217e6e5946-kube-api-access-qmcmk\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-socket-dir-parent\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-kubelet\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446594 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c8e7275-b6ac-4522-99e3-b35330ebe02a-host-slash\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.446644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446616 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cnibin\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446639 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446795 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-system-cni-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446795 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-cni-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446795 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446723 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/423f86d5-93d3-4bc3-8f97-4a52730ebfef-cni-binary-copy\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446795 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-netns\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446795 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446774 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-hostroot\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-multus-certs\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-os-release\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ec019a3-7329-4860-a941-03bc6091f53f-konnectivity-ca\") pod \"konnectivity-agent-8wghw\" (UID: \"5ec019a3-7329-4860-a941-03bc6091f53f\") " pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-cni-multus\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-os-release\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446936 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-k8s-cni-cncf-io\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-daemon-config\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.446994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.446990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-cnibin\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.447394 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447007 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-etc-kubernetes\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.447394 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rc5\" (UniqueName: \"kubernetes.io/projected/4c8e7275-b6ac-4522-99e3-b35330ebe02a-kube-api-access-x7rc5\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.447992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447659 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 21:00:25.447992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447684 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:00:25.447992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447823 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.447992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447866 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.447992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.447967 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-77hx6\"" Jul 21 21:00:25.448661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.448644 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.453379 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.452397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.453379 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.453056 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.454000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.453981 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gmfpv\"" Jul 21 21:00:25.454239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.454222 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.455877 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.455858 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.456122 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.456106 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7k5kv\"" Jul 21 21:00:25.456342 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.456327 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.458819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.458499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.460998 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.460974 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 21:00:25.461087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.461007 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 21:00:25.461087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.461080 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 21:00:25.461269 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.461253 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2rmvn\"" Jul 21 21:00:25.461484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.461465 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:25.461562 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.461544 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:25.464202 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.464183 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:25.464287 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.464265 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:25.478585 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.478561 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ld4l" Jul 21 21:00:25.489980 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.489955 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ld4l" Jul 21 21:00:25.536404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.536381 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 21:00:25.547281 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-socket-dir-parent\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547426 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-kubelet\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547426 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547320 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.547426 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547346 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa26ba2b-b138-48ae-9649-6d34fa904d64-tmp\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.547426 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547384 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7cv\" (UniqueName: \"kubernetes.io/projected/3e4e3cde-b184-49f6-bb67-1d5962d17492-kube-api-access-5r7cv\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.547426 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-kubelet\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.547426 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/423f86d5-93d3-4bc3-8f97-4a52730ebfef-cni-binary-copy\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547459 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-socket-dir-parent\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-os-release\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547521 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-kubelet\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77077877-720b-4a5b-a604-2c2b0af5ba0c-dbus\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovn-node-metrics-cert\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547644 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovnkube-script-lib\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.547673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547663 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqj5\" (UniqueName: \"kubernetes.io/projected/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-kube-api-access-nvqj5\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-cni-multus\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-os-release\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-daemon-config\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovnkube-config\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnmq\" (UniqueName: \"kubernetes.io/projected/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-kube-api-access-qvnmq\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547818 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rc5\" (UniqueName: \"kubernetes.io/projected/4c8e7275-b6ac-4522-99e3-b35330ebe02a-kube-api-access-x7rc5\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547867 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-modprobe-d\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-kubernetes\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.547986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysctl-d\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.547988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsssk\" (UniqueName: \"kubernetes.io/projected/fa26ba2b-b138-48ae-9649-6d34fa904d64-kube-api-access-dsssk\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548042 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-socket-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548108 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-tuned\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548162 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e4e3cde-b184-49f6-bb67-1d5962d17492-hosts-file\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-slash\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-log-socket\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f71e36d-9129-487a-83bf-05ce9934d058-host\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-cni-bin\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-conf-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548315 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4dr\" (UniqueName: \"kubernetes.io/projected/423f86d5-93d3-4bc3-8f97-4a52730ebfef-kube-api-access-vq4dr\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c8e7275-b6ac-4522-99e3-b35330ebe02a-iptables-alerter-script\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcmk\" (UniqueName: \"kubernetes.io/projected/3f1ce5f0-5af3-45e3-9751-43217e6e5946-kube-api-access-qmcmk\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-var-lib-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548475 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-etc-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c8e7275-b6ac-4522-99e3-b35330ebe02a-host-slash\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cnibin\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-cni-netd\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548580 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-device-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/423f86d5-93d3-4bc3-8f97-4a52730ebfef-cni-binary-copy\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-system-cni-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-cni-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-netns\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548701 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-netns\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-cni-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.548911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548884 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-os-release\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-cni-multus\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-hostroot\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.548993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-multus-certs\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-ovn\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549046 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-env-overrides\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549057 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-daemon-config\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-hostroot\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-os-release\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f71e36d-9129-487a-83bf-05ce9934d058-serviceca\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549203 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-multus-certs\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549297 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysctl-conf\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-sys\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4e3cde-b184-49f6-bb67-1d5962d17492-tmp-dir\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-registration-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.549443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ec019a3-7329-4860-a941-03bc6091f53f-konnectivity-ca\") pod \"konnectivity-agent-8wghw\" (UID: \"5ec019a3-7329-4860-a941-03bc6091f53f\") " pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549457 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-k8s-cni-cncf-io\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-systemd\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-run\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-lib-modules\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-run-ovn-kubernetes\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-cnibin\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-etc-kubernetes\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-systemd\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-var-lib-kubelet\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-node-log\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-sys-fs\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkvd\" (UniqueName: \"kubernetes.io/projected/62b19b59-3eec-4439-ab27-8b36cbb6544b-kube-api-access-qmkvd\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-system-cni-dir\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysconfig\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.550033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.549979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-host\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-run-netns\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ec019a3-7329-4860-a941-03bc6091f53f-agent-certs\") pod \"konnectivity-agent-8wghw\" (UID: \"5ec019a3-7329-4860-a941-03bc6091f53f\") " pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77077877-720b-4a5b-a604-2c2b0af5ba0c-kubelet-config\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-systemd-units\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550138 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-cni-bin\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550164 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f956x\" (UniqueName: \"kubernetes.io/projected/8f71e36d-9129-487a-83bf-05ce9934d058-kube-api-access-f956x\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.550679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.550971 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ec019a3-7329-4860-a941-03bc6091f53f-konnectivity-ca\") pod \"konnectivity-agent-8wghw\" (UID: \"5ec019a3-7329-4860-a941-03bc6091f53f\") " pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.550971 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-run-k8s-cni-cncf-io\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.550971 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550950 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-cnibin\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.551103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.550984 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-etc-kubernetes\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.551216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551194 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-system-cni-dir\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.551649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551616 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 21:00:25.551747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551663 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c8e7275-b6ac-4522-99e3-b35330ebe02a-iptables-alerter-script\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.551747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551715 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-host-var-lib-cni-bin\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.551747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551724 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-multus-conf-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.551904 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c8e7275-b6ac-4522-99e3-b35330ebe02a-host-slash\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.552017 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.551993 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f1ce5f0-5af3-45e3-9751-43217e6e5946-cnibin\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.552088 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.552021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/423f86d5-93d3-4bc3-8f97-4a52730ebfef-system-cni-dir\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.554324 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.554280 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" event={"ID":"5bb4be778fdc4d1b1c65a09ca6ed53e9","Type":"ContainerStarted","Data":"1cefe474dd47d7a61e7653bff060a39d617aa807e67356aaa05ee1cb52c898b4"} Jul 21 21:00:25.555432 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.555401 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"771a9cee6459867329caacdbd2b313c6","Type":"ContainerStarted","Data":"1267c2d554af8ad3c8be2d19e02731a32b2e6ac7dc875aa1d1d030249aae4536"} Jul 21 21:00:25.555777 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.555743 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ec019a3-7329-4860-a941-03bc6091f53f-agent-certs\") pod \"konnectivity-agent-8wghw\" (UID: \"5ec019a3-7329-4860-a941-03bc6091f53f\") " pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.557278 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.557256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rc5\" (UniqueName: \"kubernetes.io/projected/4c8e7275-b6ac-4522-99e3-b35330ebe02a-kube-api-access-x7rc5\") pod \"iptables-alerter-tlkbc\" (UID: \"4c8e7275-b6ac-4522-99e3-b35330ebe02a\") " pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.559241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.559222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4dr\" (UniqueName: \"kubernetes.io/projected/423f86d5-93d3-4bc3-8f97-4a52730ebfef-kube-api-access-vq4dr\") pod \"multus-x2hj2\" (UID: \"423f86d5-93d3-4bc3-8f97-4a52730ebfef\") " pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.560568 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.560548 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcmk\" (UniqueName: \"kubernetes.io/projected/3f1ce5f0-5af3-45e3-9751-43217e6e5946-kube-api-access-qmcmk\") pod \"multus-additional-cni-plugins-jwt58\" (UID: \"3f1ce5f0-5af3-45e3-9751-43217e6e5946\") " pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.650736 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650704 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-tuned\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.650736 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e4e3cde-b184-49f6-bb67-1d5962d17492-hosts-file\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-slash\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-log-socket\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f71e36d-9129-487a-83bf-05ce9934d058-host\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650821 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f71e36d-9129-487a-83bf-05ce9934d058-host\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650834 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-slash\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-var-lib-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650871 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-log-socket\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-etc-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650834 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e4e3cde-b184-49f6-bb67-1d5962d17492-hosts-file\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-var-lib-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-cni-netd\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650935 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-etc-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.650953 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650960 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-device-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.650990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-ovn\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-env-overrides\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651008 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-cni-netd\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651060 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-ovn\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651066 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-device-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f71e36d-9129-487a-83bf-05ce9934d058-serviceca\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysctl-conf\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-sys\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4e3cde-b184-49f6-bb67-1d5962d17492-tmp-dir\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651269 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysctl-conf\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.651233 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-sys\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651334 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-registration-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.651350 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:00:26.151329205 +0000 UTC m=+3.095950265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-registration-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.651620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651397 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-systemd\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-run\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-lib-modules\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-run-ovn-kubernetes\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-systemd\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651588 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-run\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651588 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f71e36d-9129-487a-83bf-05ce9934d058-serviceca\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-run-ovn-kubernetes\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651638 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e4e3cde-b184-49f6-bb67-1d5962d17492-tmp-dir\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651650 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-systemd\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651532 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-env-overrides\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651646 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-lib-modules\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651675 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-systemd\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-var-lib-kubelet\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-node-log\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-sys-fs\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkvd\" (UniqueName: \"kubernetes.io/projected/62b19b59-3eec-4439-ab27-8b36cbb6544b-kube-api-access-qmkvd\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.652406 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-node-log\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651804 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-var-lib-kubelet\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysconfig\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651806 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-host\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-run-netns\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651859 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-sys-fs\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651867 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysconfig\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77077877-720b-4a5b-a604-2c2b0af5ba0c-kubelet-config\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651912 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-host\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-systemd-units\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-cni-bin\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f956x\" (UniqueName: \"kubernetes.io/projected/8f71e36d-9129-487a-83bf-05ce9934d058-kube-api-access-f956x\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651975 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-systemd-units\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77077877-720b-4a5b-a604-2c2b0af5ba0c-kubelet-config\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651946 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-run-netns\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652012 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-cni-bin\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.651996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa26ba2b-b138-48ae-9649-6d34fa904d64-tmp\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7cv\" (UniqueName: \"kubernetes.io/projected/3e4e3cde-b184-49f6-bb67-1d5962d17492-kube-api-access-5r7cv\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-kubelet\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652113 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77077877-720b-4a5b-a604-2c2b0af5ba0c-dbus\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovn-node-metrics-cert\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovnkube-script-lib\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652189 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-run-openvswitch\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqj5\" (UniqueName: \"kubernetes.io/projected/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-kube-api-access-nvqj5\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652226 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-kubelet\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovnkube-config\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnmq\" (UniqueName: \"kubernetes.io/projected/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-kube-api-access-qvnmq\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652280 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-modprobe-d\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-kubernetes\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysctl-d\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsssk\" (UniqueName: \"kubernetes.io/projected/fa26ba2b-b138-48ae-9649-6d34fa904d64-kube-api-access-dsssk\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.653990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-socket-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-socket-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652783 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62b19b59-3eec-4439-ab27-8b36cbb6544b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652824 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovnkube-config\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovnkube-script-lib\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652951 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-sysctl-d\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.652974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.653000 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-kubernetes\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.653015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-modprobe-d\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.653057 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77077877-720b-4a5b-a604-2c2b0af5ba0c-dbus\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.653132 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.653201 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:00:26.153173595 +0000 UTC m=+3.097794642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.653196 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.654005 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa26ba2b-b138-48ae-9649-6d34fa904d64-etc-tuned\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.654654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.654592 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa26ba2b-b138-48ae-9649-6d34fa904d64-tmp\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.655179 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.654818 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-ovn-node-metrics-cert\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.663597 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.663569 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:25.663688 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.663602 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:25.663688 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.663620 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:25.663768 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:25.663704 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:00:26.163690265 +0000 UTC m=+3.108311324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:25.665298 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.665257 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkvd\" (UniqueName: \"kubernetes.io/projected/62b19b59-3eec-4439-ab27-8b36cbb6544b-kube-api-access-qmkvd\") pod \"aws-ebs-csi-driver-node-mfsp6\" (UID: \"62b19b59-3eec-4439-ab27-8b36cbb6544b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:25.666465 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.666442 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f956x\" (UniqueName: \"kubernetes.io/projected/8f71e36d-9129-487a-83bf-05ce9934d058-kube-api-access-f956x\") pod \"node-ca-7cgd8\" (UID: \"8f71e36d-9129-487a-83bf-05ce9934d058\") " pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.666751 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.666733 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnmq\" (UniqueName: \"kubernetes.io/projected/7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24-kube-api-access-qvnmq\") pod \"ovnkube-node-hph6g\" (UID: \"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24\") " pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.667176 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.667150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsssk\" (UniqueName: \"kubernetes.io/projected/fa26ba2b-b138-48ae-9649-6d34fa904d64-kube-api-access-dsssk\") pod \"tuned-jq5pf\" (UID: \"fa26ba2b-b138-48ae-9649-6d34fa904d64\") " pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.667405 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.667381 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7cv\" (UniqueName: \"kubernetes.io/projected/3e4e3cde-b184-49f6-bb67-1d5962d17492-kube-api-access-5r7cv\") pod \"node-resolver-wk9cv\" (UID: \"3e4e3cde-b184-49f6-bb67-1d5962d17492\") " pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.667456 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.667429 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqj5\" (UniqueName: \"kubernetes.io/projected/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-kube-api-access-nvqj5\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:25.681655 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.681590 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:25.743909 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.743868 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x2hj2" Jul 21 21:00:25.753682 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.753656 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlkbc" Jul 21 21:00:25.761531 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.761504 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:25.773035 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.773013 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jwt58" Jul 21 21:00:25.778198 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.778179 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:25.785296 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.785273 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7cgd8" Jul 21 21:00:25.791926 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.791908 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" Jul 21 21:00:25.797660 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.797642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wk9cv" Jul 21 21:00:25.802548 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:25.802529 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" Jul 21 21:00:26.158382 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.158306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:26.158382 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.158365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:26.158581 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.158489 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:26.158581 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.158506 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:26.158581 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.158551 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:00:27.158536476 +0000 UTC m=+4.103157520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:26.158685 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.158583 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:00:27.158559541 +0000 UTC m=+4.103180599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:26.259023 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.258987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:26.259180 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.259152 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:26.259180 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.259173 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:26.259180 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.259182 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:26.259327 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:26.259235 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:00:27.259221888 +0000 UTC m=+4.203842932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:26.492768 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.492692 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 20:55:26 +0000 UTC" deadline="2028-03-18 18:57:03.3041721 +0000 UTC" Jul 21 21:00:26.492768 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.492723 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14541h56m36.811453113s" Jul 21 21:00:26.492768 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.492697 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec019a3_7329_4860_a941_03bc6091f53f.slice/crio-8827cb8f494ed5f4cba08022e94744c7aaffbd8c096ba6802fc980f6fa4d4e49 WatchSource:0}: Error finding container 8827cb8f494ed5f4cba08022e94744c7aaffbd8c096ba6802fc980f6fa4d4e49: Status 404 returned error can't find the container with id 8827cb8f494ed5f4cba08022e94744c7aaffbd8c096ba6802fc980f6fa4d4e49 Jul 21 21:00:26.493863 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.493840 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4e3cde_b184_49f6_bb67_1d5962d17492.slice/crio-b046bce82ee0be648000bb7f4a034bc53ea1d4e88faa8aa3826390ce8c072ea3 WatchSource:0}: Error finding container b046bce82ee0be648000bb7f4a034bc53ea1d4e88faa8aa3826390ce8c072ea3: Status 404 returned error can't find the container with id b046bce82ee0be648000bb7f4a034bc53ea1d4e88faa8aa3826390ce8c072ea3 Jul 21 21:00:26.494605 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.494450 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423f86d5_93d3_4bc3_8f97_4a52730ebfef.slice/crio-8eb7b6b6bebfa4ec06687f305d1ae308b0d6e481b3dee14d7a5f14556bd48d3e WatchSource:0}: Error finding container 8eb7b6b6bebfa4ec06687f305d1ae308b0d6e481b3dee14d7a5f14556bd48d3e: Status 404 returned error can't find the container with id 8eb7b6b6bebfa4ec06687f305d1ae308b0d6e481b3dee14d7a5f14556bd48d3e Jul 21 21:00:26.497715 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.497685 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa26ba2b_b138_48ae_9649_6d34fa904d64.slice/crio-064abcaed4ad374bfd8e141003f0e5153d935a22cb57e50802ed8496f58f2f1b WatchSource:0}: Error finding container 064abcaed4ad374bfd8e141003f0e5153d935a22cb57e50802ed8496f58f2f1b: Status 404 returned error can't find the container with id 064abcaed4ad374bfd8e141003f0e5153d935a22cb57e50802ed8496f58f2f1b Jul 21 21:00:26.498602 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.498576 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b19b59_3eec_4439_ab27_8b36cbb6544b.slice/crio-7adc2b2550c0a55941188c31e2e3196de107e8eef2f359dd4443a93c183d6025 WatchSource:0}: Error finding container 7adc2b2550c0a55941188c31e2e3196de107e8eef2f359dd4443a93c183d6025: Status 404 returned error can't find the container with id 7adc2b2550c0a55941188c31e2e3196de107e8eef2f359dd4443a93c183d6025 Jul 21 21:00:26.499403 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.499347 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1ce5f0_5af3_45e3_9751_43217e6e5946.slice/crio-d4769e2d1926a563e70217be4c1536b2932eb7c70842396995708e129ec80fed WatchSource:0}: Error finding container d4769e2d1926a563e70217be4c1536b2932eb7c70842396995708e129ec80fed: Status 404 returned error can't find the container with id d4769e2d1926a563e70217be4c1536b2932eb7c70842396995708e129ec80fed Jul 21 21:00:26.500983 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.500391 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9bcdc9_0dfa_4a26_98c6_30f5a314ea24.slice/crio-503855741a7737dc6ddd8689ec87f1107ff32e3f4ad7abd0e27dcb4398be1db2 WatchSource:0}: Error finding container 503855741a7737dc6ddd8689ec87f1107ff32e3f4ad7abd0e27dcb4398be1db2: Status 404 returned error can't find the container with id 503855741a7737dc6ddd8689ec87f1107ff32e3f4ad7abd0e27dcb4398be1db2 Jul 21 21:00:26.502049 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.502026 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f71e36d_9129_487a_83bf_05ce9934d058.slice/crio-0d0057d2385d93c4c9a8b8c18237656b78d83f9acfa131c0868b5be79dbfcb1e WatchSource:0}: Error finding container 0d0057d2385d93c4c9a8b8c18237656b78d83f9acfa131c0868b5be79dbfcb1e: Status 404 returned error can't find the container with id 0d0057d2385d93c4c9a8b8c18237656b78d83f9acfa131c0868b5be79dbfcb1e Jul 21 21:00:26.503475 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:26.503325 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8e7275_b6ac_4522_99e3_b35330ebe02a.slice/crio-7deb4c73fd8e166cf2281ccd2e198b6f8aac375d6ddc95ff22b22f13c6da42d5 WatchSource:0}: Error finding container 7deb4c73fd8e166cf2281ccd2e198b6f8aac375d6ddc95ff22b22f13c6da42d5: Status 404 returned error can't find the container with id 7deb4c73fd8e166cf2281ccd2e198b6f8aac375d6ddc95ff22b22f13c6da42d5 Jul 21 21:00:26.561715 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.561682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7cgd8" event={"ID":"8f71e36d-9129-487a-83bf-05ce9934d058","Type":"ContainerStarted","Data":"0d0057d2385d93c4c9a8b8c18237656b78d83f9acfa131c0868b5be79dbfcb1e"} Jul 21 21:00:26.562984 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.562954 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" event={"ID":"62b19b59-3eec-4439-ab27-8b36cbb6544b","Type":"ContainerStarted","Data":"7adc2b2550c0a55941188c31e2e3196de107e8eef2f359dd4443a93c183d6025"} Jul 21 21:00:26.564078 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.564055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" event={"ID":"fa26ba2b-b138-48ae-9649-6d34fa904d64","Type":"ContainerStarted","Data":"064abcaed4ad374bfd8e141003f0e5153d935a22cb57e50802ed8496f58f2f1b"} Jul 21 21:00:26.566395 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.566374 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2hj2" event={"ID":"423f86d5-93d3-4bc3-8f97-4a52730ebfef","Type":"ContainerStarted","Data":"8eb7b6b6bebfa4ec06687f305d1ae308b0d6e481b3dee14d7a5f14556bd48d3e"} Jul 21 21:00:26.569634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.569611 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlkbc" event={"ID":"4c8e7275-b6ac-4522-99e3-b35330ebe02a","Type":"ContainerStarted","Data":"7deb4c73fd8e166cf2281ccd2e198b6f8aac375d6ddc95ff22b22f13c6da42d5"} Jul 21 21:00:26.571765 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.571741 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"503855741a7737dc6ddd8689ec87f1107ff32e3f4ad7abd0e27dcb4398be1db2"} Jul 21 21:00:26.572882 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.572862 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerStarted","Data":"d4769e2d1926a563e70217be4c1536b2932eb7c70842396995708e129ec80fed"} Jul 21 21:00:26.573964 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.573942 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wk9cv" event={"ID":"3e4e3cde-b184-49f6-bb67-1d5962d17492","Type":"ContainerStarted","Data":"b046bce82ee0be648000bb7f4a034bc53ea1d4e88faa8aa3826390ce8c072ea3"} Jul 21 21:00:26.575127 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:26.575104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8wghw" event={"ID":"5ec019a3-7329-4860-a941-03bc6091f53f","Type":"ContainerStarted","Data":"8827cb8f494ed5f4cba08022e94744c7aaffbd8c096ba6802fc980f6fa4d4e49"} Jul 21 21:00:27.166156 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.165815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:27.166156 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.166114 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:27.166382 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.166170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:27.166382 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.166196 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:00:29.166173403 +0000 UTC m=+6.110794470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:27.166382 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.166265 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:27.166382 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.166320 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:00:29.166303684 +0000 UTC m=+6.110924744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:27.267536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.267493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:27.267730 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.267699 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:27.267730 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.267720 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:27.267867 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.267733 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:27.267867 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.267793 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:00:29.26777433 +0000 UTC m=+6.212395378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:27.493582 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.493450 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 20:55:26 +0000 UTC" deadline="2028-03-25 00:06:37.334629836 +0000 UTC" Jul 21 21:00:27.493582 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.493519 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14691h6m9.841115985s" Jul 21 21:00:27.553922 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.553830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:27.554082 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.553972 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:27.554082 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.554055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:27.554221 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.554126 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:27.554221 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.554182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:27.554326 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:27.554253 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:27.608540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.608507 2570 generic.go:356] "Generic (PLEG): container finished" podID="771a9cee6459867329caacdbd2b313c6" containerID="a5a5c92f3b99bbd733074a3e7096091d4ae9913069ba4d8c7afe309017072558" exitCode=0 Jul 21 21:00:27.608885 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.608795 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"771a9cee6459867329caacdbd2b313c6","Type":"ContainerDied","Data":"a5a5c92f3b99bbd733074a3e7096091d4ae9913069ba4d8c7afe309017072558"} Jul 21 21:00:27.619783 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:27.619754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" event={"ID":"5bb4be778fdc4d1b1c65a09ca6ed53e9","Type":"ContainerStarted","Data":"41e616b8003bff0081d51095a15b81199e81b9ec8148292aa73c2dc891f006ac"} Jul 21 21:00:28.639600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:28.639479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"771a9cee6459867329caacdbd2b313c6","Type":"ContainerStarted","Data":"db30f4d169b997072ac1be6a7b1f6490be2e647f87f81fddd78f6b6d03cdffee"} Jul 21 21:00:28.659903 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:28.659049 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" podStartSLOduration=3.659024837 podStartE2EDuration="3.659024837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:27.646468744 +0000 UTC m=+4.591089814" watchObservedRunningTime="2026-07-21 21:00:28.659024837 +0000 UTC m=+5.603645904" Jul 21 21:00:29.183728 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:29.182797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:29.183728 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:29.182898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:29.183728 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.183123 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:29.183728 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.183197 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:00:33.183175257 +0000 UTC m=+10.127796314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:29.183728 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.183297 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:29.183728 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.183340 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:00:33.183325842 +0000 UTC m=+10.127947120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:29.284229 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:29.284192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:29.284428 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.284362 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:29.284428 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.284382 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:29.284428 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.284394 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:29.284629 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.284518 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:00:33.284495141 +0000 UTC m=+10.229116201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:29.550743 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:29.550583 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:29.550743 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:29.550601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:29.550743 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.550719 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:29.551346 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:29.551092 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:29.551346 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.551206 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:29.551346 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:29.551306 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:31.553175 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:31.551498 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:31.553175 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:31.551544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:31.553175 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:31.551635 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:31.553175 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:31.551716 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:31.553175 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:31.551812 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:31.553175 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:31.551875 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:33.229199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:33.228978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:33.229199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:33.229058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:33.229199 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.229160 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:33.229708 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.229233 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:00:41.229216938 +0000 UTC m=+18.173837982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:33.229708 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.229167 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:33.229708 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.229461 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:00:41.229444465 +0000 UTC m=+18.174065514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:33.330165 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:33.330118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:33.330444 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.330415 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:33.330444 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.330451 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:33.330815 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.330465 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:33.330815 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.330538 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:00:41.330516627 +0000 UTC m=+18.275137676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:33.552009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:33.551929 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:33.552162 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.552058 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:33.552494 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:33.552470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:33.552645 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.552570 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:33.552715 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:33.552700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:33.552820 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:33.552800 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:35.551662 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:35.550912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:35.551662 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:35.551050 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:35.551662 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:35.551454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:35.551662 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:35.551529 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:35.551662 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:35.551581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:35.551662 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:35.551635 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:37.550948 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:37.550915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:37.551394 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:37.550916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:37.551394 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:37.551023 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:37.551394 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:37.551099 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:37.551394 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:37.550915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:37.551394 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:37.551204 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:39.550633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:39.550600 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:39.551021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:39.550639 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:39.551021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:39.550605 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:39.551021 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:39.550738 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:39.551021 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:39.550809 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:39.551021 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:39.550875 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:41.283918 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:41.283887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:41.284408 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:41.283930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:41.284408 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.284041 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:41.284408 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.284041 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:41.284408 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.284101 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:00:57.284083005 +0000 UTC m=+34.228704069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:41.284408 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.284116 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:00:57.284109253 +0000 UTC m=+34.228730297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:41.384670 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:41.384640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:41.384869 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.384771 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:41.384869 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.384789 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:41.384869 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.384798 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:41.384984 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.384874 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:00:57.384835074 +0000 UTC m=+34.329456117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:41.551190 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:41.551111 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:41.551190 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:41.551184 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:41.551417 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.551274 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:41.551417 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:41.551182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:41.551417 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.551388 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:41.551534 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:41.551463 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:43.551761 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:43.551728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:43.551761 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:43.551765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:43.552284 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:43.551839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:43.552284 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:43.551861 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:43.552284 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:43.551912 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:43.552284 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:43.551997 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:45.550587 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:45.550555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:45.551054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:45.550552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:45.551054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:45.550691 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:45.551054 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:45.550684 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:45.551054 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:45.550769 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:45.551054 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:45.550862 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:46.681436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.680992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x2hj2" event={"ID":"423f86d5-93d3-4bc3-8f97-4a52730ebfef","Type":"ContainerStarted","Data":"ce373efed3f20f33b3dced84410947bed8944e1e901a59fd3e1657dd831e6fba"} Jul 21 21:00:46.683779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.683755 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"f8026af18f34e80834e59cd81e80a733632913418f831926c58af90574c8bcae"} Jul 21 21:00:46.683886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.683821 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"7ed24f87d1600196699f36410803575ae20faaef04fd4e32f14253de0c0eb1b9"} Jul 21 21:00:46.683886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.683857 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"4a086b45aceea4a509bddc3e7b649cb4026ff93b7b9cf9e1f88f5de4bb5e580e"} Jul 21 21:00:46.683886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.683886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"3521af2b92b7408018badc21105f2940093ca643c60afa5dc0e77e88d553fe3c"} Jul 21 21:00:46.684027 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.683919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"0483f615f1edbea3d4e5fe047130280909e1b38cca17ca27ebf32a4cd3d46c69"} Jul 21 21:00:46.685249 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.685230 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f1ce5f0-5af3-45e3-9751-43217e6e5946" containerID="59165cc34d358146106ebd763770afc63ce2f0c2c61eb851fd7e4ce88e5cffa0" exitCode=0 Jul 21 21:00:46.685347 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.685300 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerDied","Data":"59165cc34d358146106ebd763770afc63ce2f0c2c61eb851fd7e4ce88e5cffa0"} Jul 21 21:00:46.688906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.687579 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wk9cv" event={"ID":"3e4e3cde-b184-49f6-bb67-1d5962d17492","Type":"ContainerStarted","Data":"cc27d241216ab31a8d91f8256f3e78b527700ea81b15c93c3efa3b559455afc7"} Jul 21 21:00:46.689997 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.689977 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8wghw" event={"ID":"5ec019a3-7329-4860-a941-03bc6091f53f","Type":"ContainerStarted","Data":"45495d839d5de1c99f73385985ced736522ea4a7048f4d2f17a6bbe4aa9935b7"} Jul 21 21:00:46.691346 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.691327 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7cgd8" event={"ID":"8f71e36d-9129-487a-83bf-05ce9934d058","Type":"ContainerStarted","Data":"a6d660a802607d4b0c40b406957d41b44844d37a4b8f604de8a7637ab5c86fc1"} Jul 21 21:00:46.692608 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.692588 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" event={"ID":"62b19b59-3eec-4439-ab27-8b36cbb6544b","Type":"ContainerStarted","Data":"2bebe0796eff0f6e352dd95ba1017baf19b4fd7ed713516d24b59480f6d2f25f"} Jul 21 21:00:46.694167 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.694149 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" event={"ID":"fa26ba2b-b138-48ae-9649-6d34fa904d64","Type":"ContainerStarted","Data":"31160575220a8e0c2f84177ca09cb4a641f76af9a6582f010ffe8369029b9e7e"} Jul 21 21:00:46.698797 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.698762 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" podStartSLOduration=21.698749752 podStartE2EDuration="21.698749752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:28.659797513 +0000 UTC m=+5.604418584" watchObservedRunningTime="2026-07-21 21:00:46.698749752 +0000 UTC m=+23.643370818" Jul 21 21:00:46.699196 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.699172 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x2hj2" podStartSLOduration=3.126236133 podStartE2EDuration="22.699164346s" totalImagesPullingTime="19.572928213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:46.698610725 +0000 UTC m=+23.643231791" watchObservedRunningTime="2026-07-21 21:00:46.699164346 +0000 UTC m=+23.643785444" Jul 21 21:00:46.713272 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.713234 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7cgd8" podStartSLOduration=3.232321503 podStartE2EDuration="22.71322367s" totalImagesPullingTime="19.480902167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:46.713017067 +0000 UTC m=+23.657638142" watchObservedRunningTime="2026-07-21 21:00:46.71322367 +0000 UTC m=+23.657844735" Jul 21 21:00:46.729731 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.729695 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jq5pf" podStartSLOduration=3.201847145 podStartE2EDuration="22.729683912s" totalImagesPullingTime="19.527836767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:46.729198233 +0000 UTC m=+23.673819299" watchObservedRunningTime="2026-07-21 21:00:46.729683912 +0000 UTC m=+23.674304977" Jul 21 21:00:46.742995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.742945 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8wghw" podStartSLOduration=6.943423155 podStartE2EDuration="22.742929435s" totalImagesPullingTime="15.79950628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:46.742898628 +0000 UTC m=+23.687519694" watchObservedRunningTime="2026-07-21 21:00:46.742929435 +0000 UTC m=+23.687550498" Jul 21 21:00:46.761078 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:46.761037 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wk9cv" podStartSLOduration=3.24355616 podStartE2EDuration="22.761025151s" totalImagesPullingTime="19.517468991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:46.760706367 +0000 UTC m=+23.705327433" watchObservedRunningTime="2026-07-21 21:00:46.761025151 +0000 UTC m=+23.705646215" Jul 21 21:00:47.550274 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.550247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:47.550448 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.550250 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:47.550448 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:47.550432 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:47.550564 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.550475 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:47.550616 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:47.550570 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:47.550672 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:47.550655 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:47.650349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.650321 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 21:00:47.697052 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.697014 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlkbc" event={"ID":"4c8e7275-b6ac-4522-99e3-b35330ebe02a","Type":"ContainerStarted","Data":"c59230bfed3067d0b13956e7d86ddf661089f10773515dbb62cee2fe1cd11c97"} Jul 21 21:00:47.700204 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.700175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"c479ee52a797485513cf154b2ee47f2007e040bd15d5fcb5ad9eb1fd1ae90227"} Jul 21 21:00:47.703635 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.703613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" event={"ID":"62b19b59-3eec-4439-ab27-8b36cbb6544b","Type":"ContainerStarted","Data":"725962b33103eb60534095c1c01081e3eaa2deb1453fa3858538b8bb71c02fbe"} Jul 21 21:00:47.714046 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:47.714008 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tlkbc" podStartSLOduration=4.232887046 podStartE2EDuration="23.71399569s" totalImagesPullingTime="19.481108644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:47.713861209 +0000 UTC m=+24.658482274" watchObservedRunningTime="2026-07-21 21:00:47.71399569 +0000 UTC m=+24.658616755" Jul 21 21:00:48.523705 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:48.523592 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T21:00:47.650348237Z","UUID":"6868467b-065b-4d62-969c-1836f2d0f68d","Handler":null,"Name":"","Endpoint":""} Jul 21 21:00:48.525061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:48.525040 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 21:00:48.525191 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:48.525076 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 21:00:49.550810 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.550782 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:49.551146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.550782 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:49.551146 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:49.550898 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:49.551146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.550913 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:49.551146 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:49.551000 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:49.551146 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:49.551089 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:49.568450 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.568403 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:49.569044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.569022 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:49.712783 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.712747 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"c4b9fd7dcecaa8ce358796c90849fcd3895644e13a9a2c82a072d888330e1d74"} Jul 21 21:00:49.715186 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.715153 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" event={"ID":"62b19b59-3eec-4439-ab27-8b36cbb6544b","Type":"ContainerStarted","Data":"48e127944d4947b5d08820ee71ead908b40fefb997fd708f1eb476f28a4ec3d9"} Jul 21 21:00:49.715473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.715448 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:49.715902 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.715886 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8wghw" Jul 21 21:00:49.736547 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:49.736510 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfsp6" podStartSLOduration=3.077008447 podStartE2EDuration="25.736497629s" totalImagesPullingTime="22.659489182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:49.736471161 +0000 UTC m=+26.681092247" watchObservedRunningTime="2026-07-21 21:00:49.736497629 +0000 UTC m=+26.681118695" Jul 21 21:00:51.551575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:51.551324 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:51.552151 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:51.551655 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:51.552151 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:51.551470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:51.552151 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:51.551722 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:51.552151 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:51.551411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:51.552151 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:51.551778 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:52.725344 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:52.725166 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" event={"ID":"7e9bcdc9-0dfa-4a26-98c6-30f5a314ea24","Type":"ContainerStarted","Data":"acb3351181f4a744e571618b9e4ca2639a50f62ea582de8e0f0ca0df04671e7c"} Jul 21 21:00:52.733962 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:52.733887 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:52.734031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:52.733965 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:52.753350 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:52.753313 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:52.767449 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:52.767406 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" podStartSLOduration=9.192227031 podStartE2EDuration="28.767387999s" totalImagesPullingTime="19.575160968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:52.767253299 +0000 UTC m=+29.711874363" watchObservedRunningTime="2026-07-21 21:00:52.767387999 +0000 UTC m=+29.712009063" Jul 21 21:00:53.551512 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.551465 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:53.551766 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.551568 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:53.551766 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.551613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:53.551766 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:53.551630 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:53.551766 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:53.551686 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:53.551973 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:53.551768 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:53.729403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.729344 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f1ce5f0-5af3-45e3-9751-43217e6e5946" containerID="624a7b5fb6ba16339364535986907aed2db6e805c9144e1f24116702cf7de98c" exitCode=0 Jul 21 21:00:53.729403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.729402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerDied","Data":"624a7b5fb6ba16339364535986907aed2db6e805c9144e1f24116702cf7de98c"} Jul 21 21:00:53.733228 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.731537 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:53.761563 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:53.761525 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:00:54.476567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:54.476532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ddkj"] Jul 21 21:00:54.476753 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:54.476704 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:54.476844 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:54.476812 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:54.477299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:54.477280 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpdk5"] Jul 21 21:00:54.477388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:54.477378 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:54.477476 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:54.477449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:54.479853 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:54.479815 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twf9r"] Jul 21 21:00:54.479986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:54.479927 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:54.480082 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:54.480064 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:55.735854 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:55.735678 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f1ce5f0-5af3-45e3-9751-43217e6e5946" containerID="b176395972e432a0c3ccd60a86c751ff5d2ef7c6a61e267daf01076471244318" exitCode=0 Jul 21 21:00:55.736316 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:55.735758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerDied","Data":"b176395972e432a0c3ccd60a86c751ff5d2ef7c6a61e267daf01076471244318"} Jul 21 21:00:56.550881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:56.550856 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:56.550995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:56.550878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:56.550995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:56.550954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:56.550995 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:56.550957 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:56.551085 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:56.551030 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:56.551122 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:56.551104 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:57.297081 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:57.297043 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:57.297658 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:57.297109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:57.297658 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.297213 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:57.297658 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.297216 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:57.297658 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.297271 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret podName:77077877-720b-4a5b-a604-2c2b0af5ba0c nodeName:}" failed. No retries permitted until 2026-07-21 21:01:29.297254323 +0000 UTC m=+66.241875369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret") pod "global-pull-secret-syncer-6ddkj" (UID: "77077877-720b-4a5b-a604-2c2b0af5ba0c") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:57.297658 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.297291 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:01:29.297281007 +0000 UTC m=+66.241902053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:57.398258 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:57.398189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:57.398409 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.398324 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:57.398409 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.398338 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:57.398409 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.398347 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wq285 for pod openshift-network-diagnostics/network-check-target-lpdk5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:57.398511 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:57.398420 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285 podName:44012990-eac1-4537-b71c-a9d6b8d16a8b nodeName:}" failed. No retries permitted until 2026-07-21 21:01:29.398406894 +0000 UTC m=+66.343027939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq285" (UniqueName: "kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285") pod "network-check-target-lpdk5" (UID: "44012990-eac1-4537-b71c-a9d6b8d16a8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:57.457540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:57.457503 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:00:57.743001 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:57.742960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerStarted","Data":"a84a56795428650b3f9d22487fb97cf8acaa2c1cfa45ccc47f722347c86e3658"} Jul 21 21:00:58.550610 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.550155 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:00:58.550610 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.550179 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:00:58.550610 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.550264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:00:58.550610 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:58.550267 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpdk5" podUID="44012990-eac1-4537-b71c-a9d6b8d16a8b" Jul 21 21:00:58.550610 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:58.550340 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ddkj" podUID="77077877-720b-4a5b-a604-2c2b0af5ba0c" Jul 21 21:00:58.550610 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:58.550457 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-twf9r" podUID="2742e7c0-42ac-4ed5-89dc-e77b9fb2381d" Jul 21 21:00:58.748126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.748043 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f1ce5f0-5af3-45e3-9751-43217e6e5946" containerID="a84a56795428650b3f9d22487fb97cf8acaa2c1cfa45ccc47f722347c86e3658" exitCode=0 Jul 21 21:00:58.748126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.748100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerDied","Data":"a84a56795428650b3f9d22487fb97cf8acaa2c1cfa45ccc47f722347c86e3658"} Jul 21 21:00:58.840886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.840856 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeReady" Jul 21 21:00:58.841060 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.841002 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 21:00:58.895515 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.895483 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rk9dq"] Jul 21 21:00:58.919629 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.919600 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5qq9n"] Jul 21 21:00:58.919756 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.919725 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:58.929380 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.929343 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 21:00:58.929380 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.929352 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wp8nm\"" Jul 21 21:00:58.935713 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.935695 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rk9dq"] Jul 21 21:00:58.935819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.935738 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5qq9n"] Jul 21 21:00:58.935819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.935809 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:58.969570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.969547 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 21:00:58.969896 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.969876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 21:00:58.969996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.969876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 21:00:58.969996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.969882 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 21:00:58.970203 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:58.970188 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bv9cb\"" Jul 21 21:00:59.012138 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.012066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.012138 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.012119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe43a134-8bcb-41ac-8b93-f9d7f657f094-tmp-dir\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.012333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.012144 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe43a134-8bcb-41ac-8b93-f9d7f657f094-config-volume\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.012333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.012169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97zj\" (UniqueName: \"kubernetes.io/projected/666656f7-3729-44fa-a480-554e40f6e25c-kube-api-access-k97zj\") pod \"ingress-canary-5qq9n\" (UID: \"666656f7-3729-44fa-a480-554e40f6e25c\") " pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.012333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.012255 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/666656f7-3729-44fa-a480-554e40f6e25c-cert\") pod \"ingress-canary-5qq9n\" (UID: \"666656f7-3729-44fa-a480-554e40f6e25c\") " pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.012333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.012327 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jbv8\" (UniqueName: \"kubernetes.io/projected/fe43a134-8bcb-41ac-8b93-f9d7f657f094-kube-api-access-8jbv8\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.113214 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113186 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/666656f7-3729-44fa-a480-554e40f6e25c-cert\") pod \"ingress-canary-5qq9n\" (UID: \"666656f7-3729-44fa-a480-554e40f6e25c\") " pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.113419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jbv8\" (UniqueName: \"kubernetes.io/projected/fe43a134-8bcb-41ac-8b93-f9d7f657f094-kube-api-access-8jbv8\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.113419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.113419 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:59.113352 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:00:59.113419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113412 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe43a134-8bcb-41ac-8b93-f9d7f657f094-tmp-dir\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.113636 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:59.113428 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:59.61340958 +0000 UTC m=+36.558030636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:00:59.113636 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe43a134-8bcb-41ac-8b93-f9d7f657f094-config-volume\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.113636 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97zj\" (UniqueName: \"kubernetes.io/projected/666656f7-3729-44fa-a480-554e40f6e25c-kube-api-access-k97zj\") pod \"ingress-canary-5qq9n\" (UID: \"666656f7-3729-44fa-a480-554e40f6e25c\") " pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.113910 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.113892 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe43a134-8bcb-41ac-8b93-f9d7f657f094-tmp-dir\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.114170 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.114151 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe43a134-8bcb-41ac-8b93-f9d7f657f094-config-volume\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.118402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.118354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/666656f7-3729-44fa-a480-554e40f6e25c-cert\") pod \"ingress-canary-5qq9n\" (UID: \"666656f7-3729-44fa-a480-554e40f6e25c\") " pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.123265 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.123239 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97zj\" (UniqueName: \"kubernetes.io/projected/666656f7-3729-44fa-a480-554e40f6e25c-kube-api-access-k97zj\") pod \"ingress-canary-5qq9n\" (UID: \"666656f7-3729-44fa-a480-554e40f6e25c\") " pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.123721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.123701 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jbv8\" (UniqueName: \"kubernetes.io/projected/fe43a134-8bcb-41ac-8b93-f9d7f657f094-kube-api-access-8jbv8\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.245697 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.245663 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5qq9n" Jul 21 21:00:59.454114 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.453913 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5qq9n"] Jul 21 21:00:59.458322 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:00:59.458295 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666656f7_3729_44fa_a480_554e40f6e25c.slice/crio-3224162fea65deff941075f0c09aa0465f610dff46e611288a2fb164715c8cd6 WatchSource:0}: Error finding container 3224162fea65deff941075f0c09aa0465f610dff46e611288a2fb164715c8cd6: Status 404 returned error can't find the container with id 3224162fea65deff941075f0c09aa0465f610dff46e611288a2fb164715c8cd6 Jul 21 21:00:59.617816 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.617705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:00:59.618300 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:59.617844 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:00:59.618300 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:00:59.617914 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:00.617892826 +0000 UTC m=+37.562513870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:00:59.751619 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:00:59.751577 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5qq9n" event={"ID":"666656f7-3729-44fa-a480-554e40f6e25c","Type":"ContainerStarted","Data":"3224162fea65deff941075f0c09aa0465f610dff46e611288a2fb164715c8cd6"} Jul 21 21:01:00.550554 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.550521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:01:00.550554 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.550547 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:01:00.550799 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.550638 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:01:00.553685 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.553661 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvwws\"" Jul 21 21:01:00.553817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.553661 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:01:00.554831 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.554802 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:01:00.555116 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.555094 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:01:00.555116 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.555112 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slzpm\"" Jul 21 21:01:00.555252 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.555238 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:01:00.610559 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.610534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:00.625570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.625545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:01:00.625940 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:00.625687 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:00.625940 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:00.625758 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:02.625736699 +0000 UTC m=+39.570357766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:01:00.835913 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.835832 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:00.919983 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:00.919904 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:02.266291 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:02.266258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:02.332986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:02.332948 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:02.642356 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:02.642271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:01:02.642515 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:02.642442 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:02.642515 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:02.642509 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:06.642493858 +0000 UTC m=+43.587114900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:01:03.763585 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:03.763417 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f1ce5f0-5af3-45e3-9751-43217e6e5946" containerID="5dcc9f4c65ed53258666d8d462c887cdd260e5765f660c0c1d0fed8bc6244474" exitCode=0 Jul 21 21:01:03.763987 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:03.763471 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerDied","Data":"5dcc9f4c65ed53258666d8d462c887cdd260e5765f660c0c1d0fed8bc6244474"} Jul 21 21:01:03.768054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:03.768034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5qq9n" event={"ID":"666656f7-3729-44fa-a480-554e40f6e25c","Type":"ContainerStarted","Data":"4641b0982039d40c7cb6e7ae1e3ca8e07f71d85484c837e9f4a7bea49e05e1a9"} Jul 21 21:01:03.807607 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:03.807562 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5qq9n" podStartSLOduration=2.450183455 podStartE2EDuration="5.807543295s" totalImagesPullingTime="3.35735984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:03.806658092 +0000 UTC m=+40.751279156" watchObservedRunningTime="2026-07-21 21:01:03.807543295 +0000 UTC m=+40.752164447" Jul 21 21:01:04.089631 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:04.089594 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:04.151906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:04.151870 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:04.773551 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:04.773516 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f1ce5f0-5af3-45e3-9751-43217e6e5946" containerID="364c79156afeb7c79bec7ecf620a30137ba37c9222f8ba1ef3747181a5c7bf99" exitCode=0 Jul 21 21:01:04.773980 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:04.773606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerDied","Data":"364c79156afeb7c79bec7ecf620a30137ba37c9222f8ba1ef3747181a5c7bf99"} Jul 21 21:01:05.023192 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:05.023159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:05.118873 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:05.118802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:05.779091 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:05.779059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwt58" event={"ID":"3f1ce5f0-5af3-45e3-9751-43217e6e5946","Type":"ContainerStarted","Data":"722765a0de3ee68dee0e1c190e56cda6fecc4062bb525061fd457b35acb27ba3"} Jul 21 21:01:05.806376 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:05.806322 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jwt58" podStartSLOduration=8.442491742 podStartE2EDuration="41.806307034s" totalImagesPullingTime="33.363815292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:05.80431321 +0000 UTC m=+42.748934275" watchObservedRunningTime="2026-07-21 21:01:05.806307034 +0000 UTC m=+42.750928121" Jul 21 21:01:05.928622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:05.928592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:06.024972 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.024942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:06.126620 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.126554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:06.362018 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.361986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:06.461608 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.461573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:06.530949 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.530916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:06.673985 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.671440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:01:06.673985 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:06.671705 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:06.673985 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:06.671787 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:14.671767277 +0000 UTC m=+51.616388320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:01:06.945612 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:06.945578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:14.725629 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:14.725589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:01:14.726017 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:14.725735 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:14.726017 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:14.725808 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:30.725791117 +0000 UTC m=+67.670412162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:01:17.619179 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:17.619147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:17.692511 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:17.692488 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:21.890047 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:21.890012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:21.956709 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:21.956681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:22.103099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:22.103066 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:22.203145 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:22.203116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:22.310645 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:22.310606 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:22.407616 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:22.407592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:25.750174 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:25.750142 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hph6g" Jul 21 21:01:25.935774 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:25.935743 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:26.031914 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:26.031849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:29.322247 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.322207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:01:29.322671 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.322258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:01:29.325264 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.325246 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:01:29.325374 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.325340 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:01:29.332589 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:29.332574 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 21:01:29.332640 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:29.332629 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs podName:2742e7c0-42ac-4ed5-89dc-e77b9fb2381d nodeName:}" failed. No retries permitted until 2026-07-21 21:02:33.332612629 +0000 UTC m=+130.277233672 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs") pod "network-metrics-daemon-twf9r" (UID: "2742e7c0-42ac-4ed5-89dc-e77b9fb2381d") : secret "metrics-daemon-secret" not found Jul 21 21:01:29.336480 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.336464 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77077877-720b-4a5b-a604-2c2b0af5ba0c-original-pull-secret\") pod \"global-pull-secret-syncer-6ddkj\" (UID: \"77077877-720b-4a5b-a604-2c2b0af5ba0c\") " pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:01:29.363508 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.363488 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ddkj" Jul 21 21:01:29.422780 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.422749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:01:29.433136 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.433108 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:01:29.446439 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.446419 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:01:29.457225 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.457204 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq285\" (UniqueName: \"kubernetes.io/projected/44012990-eac1-4537-b71c-a9d6b8d16a8b-kube-api-access-wq285\") pod \"network-check-target-lpdk5\" (UID: \"44012990-eac1-4537-b71c-a9d6b8d16a8b\") " pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:01:29.485590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.485565 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ddkj"] Jul 21 21:01:29.489114 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:29.489094 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77077877_720b_4a5b_a604_2c2b0af5ba0c.slice/crio-18ca86c9a05d7940e2a919f4c8770bd70df16d9a86e5dee327cea4682baf7fae WatchSource:0}: Error finding container 18ca86c9a05d7940e2a919f4c8770bd70df16d9a86e5dee327cea4682baf7fae: Status 404 returned error can't find the container with id 18ca86c9a05d7940e2a919f4c8770bd70df16d9a86e5dee327cea4682baf7fae Jul 21 21:01:29.673714 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.673692 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slzpm\"" Jul 21 21:01:29.681402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.681382 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:01:29.797808 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.796518 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpdk5"] Jul 21 21:01:29.801587 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:29.801565 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44012990_eac1_4537_b71c_a9d6b8d16a8b.slice/crio-daedc77e4cc028597eab074d0d64f00237f6c84afb724c3e71569ec6121478d5 WatchSource:0}: Error finding container daedc77e4cc028597eab074d0d64f00237f6c84afb724c3e71569ec6121478d5: Status 404 returned error can't find the container with id daedc77e4cc028597eab074d0d64f00237f6c84afb724c3e71569ec6121478d5 Jul 21 21:01:29.832437 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.832402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpdk5" event={"ID":"44012990-eac1-4537-b71c-a9d6b8d16a8b","Type":"ContainerStarted","Data":"daedc77e4cc028597eab074d0d64f00237f6c84afb724c3e71569ec6121478d5"} Jul 21 21:01:29.833401 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:29.833379 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ddkj" event={"ID":"77077877-720b-4a5b-a604-2c2b0af5ba0c","Type":"ContainerStarted","Data":"18ca86c9a05d7940e2a919f4c8770bd70df16d9a86e5dee327cea4682baf7fae"} Jul 21 21:01:30.730780 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:30.730738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:01:30.731257 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:30.730922 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:30.731257 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:30.731024 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls podName:fe43a134-8bcb-41ac-8b93-f9d7f657f094 nodeName:}" failed. No retries permitted until 2026-07-21 21:02:02.731001012 +0000 UTC m=+99.675622056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls") pod "dns-default-rk9dq" (UID: "fe43a134-8bcb-41ac-8b93-f9d7f657f094") : secret "dns-default-metrics-tls" not found Jul 21 21:01:33.704640 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.704607 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz"] Jul 21 21:01:33.725804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.725345 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c87df747f-8jzjp"] Jul 21 21:01:33.725804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.725488 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" Jul 21 21:01:33.728613 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.728586 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 21:01:33.728955 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.728923 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:01:33.729050 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.728980 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9nkvc\"" Jul 21 21:01:33.741025 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.741004 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz"] Jul 21 21:01:33.741103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.741041 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c87df747f-8jzjp"] Jul 21 21:01:33.741103 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.741053 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.743510 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.743475 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 21:01:33.745044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.744145 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 21:01:33.745044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.744227 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 21:01:33.745044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.744325 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58h5p\"" Jul 21 21:01:33.752553 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.752512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-trusted-ca\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.752653 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.752593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvrk\" (UniqueName: \"kubernetes.io/projected/91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f-kube-api-access-rhvrk\") pod \"volume-data-source-validator-78cf4c6f6-j8hjz\" (UID: \"91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" Jul 21 21:01:33.752653 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.752624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.752769 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.752693 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-image-registry-private-configuration\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.753235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.753093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-registry-certificates\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.753235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.753127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wz4\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-kube-api-access-g2wz4\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.753235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.753154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4a7e50e-67ca-4842-a470-e80103746333-ca-trust-extracted\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.753235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.753176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-installation-pull-secrets\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.753924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.753902 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 21:01:33.753996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.753932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-bound-sa-token\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.824414 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.824382 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg"] Jul 21 21:01:33.846226 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.845337 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2rwnh"] Jul 21 21:01:33.846492 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.846469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" Jul 21 21:01:33.849071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.849052 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pmdfq\"" Jul 21 21:01:33.854248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-trusted-ca\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854345 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvrk\" (UniqueName: \"kubernetes.io/projected/91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f-kube-api-access-rhvrk\") pod \"volume-data-source-validator-78cf4c6f6-j8hjz\" (UID: \"91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" Jul 21 21:01:33.854345 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-image-registry-private-configuration\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-registry-certificates\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wz4\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-kube-api-access-g2wz4\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4a7e50e-67ca-4842-a470-e80103746333-ca-trust-extracted\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-installation-pull-secrets\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854731 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854512 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8sg\" (UniqueName: \"kubernetes.io/projected/43b06cae-7bf6-4450-b3ed-8b1384087e66-kube-api-access-qx8sg\") pod \"network-check-source-7bc5b6bf85-q9zsg\" (UID: \"43b06cae-7bf6-4450-b3ed-8b1384087e66\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" Jul 21 21:01:33.854731 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-bound-sa-token\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.854834 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:33.854790 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:01:33.854834 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:33.854805 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c87df747f-8jzjp: secret "image-registry-tls" not found Jul 21 21:01:33.854932 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:33.854854 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls podName:a4a7e50e-67ca-4842-a470-e80103746333 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:34.354837134 +0000 UTC m=+71.299458190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls") pod "image-registry-c87df747f-8jzjp" (UID: "a4a7e50e-67ca-4842-a470-e80103746333") : secret "image-registry-tls" not found Jul 21 21:01:33.854932 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.854855 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4a7e50e-67ca-4842-a470-e80103746333-ca-trust-extracted\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.855055 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.855040 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-registry-certificates\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.855539 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.855514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-trusted-ca\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.857441 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.857415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-installation-pull-secrets\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.857533 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.857519 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-image-registry-private-configuration\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.869241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.869221 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg"] Jul 21 21:01:33.869384 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.869265 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2rwnh"] Jul 21 21:01:33.869384 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.869336 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.872668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.872648 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 21:01:33.872807 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.872789 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w7jhs\"" Jul 21 21:01:33.873124 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.872979 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 21:01:33.873667 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.873499 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 21:01:33.873986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.873822 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 21:01:33.875388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.875340 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-bound-sa-token\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.875667 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.875623 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wz4\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-kube-api-access-g2wz4\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:33.876024 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.876003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvrk\" (UniqueName: \"kubernetes.io/projected/91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f-kube-api-access-rhvrk\") pod \"volume-data-source-validator-78cf4c6f6-j8hjz\" (UID: \"91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" Jul 21 21:01:33.879725 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.879702 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 21:01:33.955347 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8201-c4ba-4d11-b47a-1db1a035dda7-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.955499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955355 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6w4\" (UniqueName: \"kubernetes.io/projected/104d8201-c4ba-4d11-b47a-1db1a035dda7-kube-api-access-kf6w4\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.955499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d8201-c4ba-4d11-b47a-1db1a035dda7-snapshots\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.955499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8201-c4ba-4d11-b47a-1db1a035dda7-service-ca-bundle\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.955499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955453 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d8201-c4ba-4d11-b47a-1db1a035dda7-tmp\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.955499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8201-c4ba-4d11-b47a-1db1a035dda7-serving-cert\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:33.955730 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.955529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8sg\" (UniqueName: \"kubernetes.io/projected/43b06cae-7bf6-4450-b3ed-8b1384087e66-kube-api-access-qx8sg\") pod \"network-check-source-7bc5b6bf85-q9zsg\" (UID: \"43b06cae-7bf6-4450-b3ed-8b1384087e66\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" Jul 21 21:01:33.964303 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.964278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8sg\" (UniqueName: \"kubernetes.io/projected/43b06cae-7bf6-4450-b3ed-8b1384087e66-kube-api-access-qx8sg\") pod \"network-check-source-7bc5b6bf85-q9zsg\" (UID: \"43b06cae-7bf6-4450-b3ed-8b1384087e66\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" Jul 21 21:01:33.974538 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:33.974513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:34.036996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.036967 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" Jul 21 21:01:34.056477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8201-c4ba-4d11-b47a-1db1a035dda7-service-ca-bundle\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.056477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d8201-c4ba-4d11-b47a-1db1a035dda7-tmp\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.056688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8201-c4ba-4d11-b47a-1db1a035dda7-serving-cert\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.056688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8201-c4ba-4d11-b47a-1db1a035dda7-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.056688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056601 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6w4\" (UniqueName: \"kubernetes.io/projected/104d8201-c4ba-4d11-b47a-1db1a035dda7-kube-api-access-kf6w4\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.056688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d8201-c4ba-4d11-b47a-1db1a035dda7-snapshots\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.056939 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.056916 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104d8201-c4ba-4d11-b47a-1db1a035dda7-tmp\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.057139 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.057113 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8201-c4ba-4d11-b47a-1db1a035dda7-service-ca-bundle\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.057260 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.057236 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/104d8201-c4ba-4d11-b47a-1db1a035dda7-snapshots\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.057706 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.057685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/104d8201-c4ba-4d11-b47a-1db1a035dda7-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.059320 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.059301 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/104d8201-c4ba-4d11-b47a-1db1a035dda7-serving-cert\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.066156 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.066130 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6w4\" (UniqueName: \"kubernetes.io/projected/104d8201-c4ba-4d11-b47a-1db1a035dda7-kube-api-access-kf6w4\") pod \"insights-operator-555d9d7885-2rwnh\" (UID: \"104d8201-c4ba-4d11-b47a-1db1a035dda7\") " pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.075851 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.075824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:34.156419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.156385 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" Jul 21 21:01:34.192832 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.192807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" Jul 21 21:01:34.358521 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.358494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:34.358672 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:34.358620 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:01:34.358672 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:34.358632 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c87df747f-8jzjp: secret "image-registry-tls" not found Jul 21 21:01:34.358742 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:34.358683 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls podName:a4a7e50e-67ca-4842-a470-e80103746333 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:35.358669076 +0000 UTC m=+72.303290119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls") pod "image-registry-c87df747f-8jzjp" (UID: "a4a7e50e-67ca-4842-a470-e80103746333") : secret "image-registry-tls" not found Jul 21 21:01:34.801102 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.801074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2rwnh"] Jul 21 21:01:34.803864 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:34.803836 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104d8201_c4ba_4d11_b47a_1db1a035dda7.slice/crio-dcfd7044df830b6570d8df68fabf208d34dc7e6c51231d4a5e25133b942fbb8a WatchSource:0}: Error finding container dcfd7044df830b6570d8df68fabf208d34dc7e6c51231d4a5e25133b942fbb8a: Status 404 returned error can't find the container with id dcfd7044df830b6570d8df68fabf208d34dc7e6c51231d4a5e25133b942fbb8a Jul 21 21:01:34.850881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.850842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" event={"ID":"104d8201-c4ba-4d11-b47a-1db1a035dda7","Type":"ContainerStarted","Data":"dcfd7044df830b6570d8df68fabf208d34dc7e6c51231d4a5e25133b942fbb8a"} Jul 21 21:01:34.852640 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.852608 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpdk5" event={"ID":"44012990-eac1-4537-b71c-a9d6b8d16a8b","Type":"ContainerStarted","Data":"d6e4f3647167d54cbdee332e8a28add5d926abb2ecd3d4ec5a8f158275c1bffe"} Jul 21 21:01:34.852832 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.852739 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:01:34.854122 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.854096 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ddkj" event={"ID":"77077877-720b-4a5b-a604-2c2b0af5ba0c","Type":"ContainerStarted","Data":"1188c9b3e33dfb9dade0fb08d3fa7b4e13583132ee623caa8600fcf071479e2a"} Jul 21 21:01:34.871581 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:34.871163 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lpdk5" podStartSLOduration=66.023257626 podStartE2EDuration="1m10.871144998s" totalImagesPullingTime="4.847887372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:34.870409046 +0000 UTC m=+71.815030112" watchObservedRunningTime="2026-07-21 21:01:34.871144998 +0000 UTC m=+71.815766063" Jul 21 21:01:35.018947 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.018745 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6ddkj" podStartSLOduration=65.843330736 podStartE2EDuration="1m11.018726775s" totalImagesPullingTime="5.175396039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:34.885691865 +0000 UTC m=+71.830312930" watchObservedRunningTime="2026-07-21 21:01:35.018726775 +0000 UTC m=+71.963347866" Jul 21 21:01:35.019134 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.019120 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz"] Jul 21 21:01:35.021581 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:35.021555 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f8a9a5_9cd0_4c99_ae2f_cc2c0c09fe8f.slice/crio-22541eb6fe2a0247746f968337928d4531b2cc1e27d8e95f56246ec1730c2c0b WatchSource:0}: Error finding container 22541eb6fe2a0247746f968337928d4531b2cc1e27d8e95f56246ec1730c2c0b: Status 404 returned error can't find the container with id 22541eb6fe2a0247746f968337928d4531b2cc1e27d8e95f56246ec1730c2c0b Jul 21 21:01:35.021798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.021779 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg"] Jul 21 21:01:35.024626 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:35.024602 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b06cae_7bf6_4450_b3ed_8b1384087e66.slice/crio-f231b31744a99fbc56edcc6af00bce4b989872b9c796331935ab6e3208bd6f48 WatchSource:0}: Error finding container f231b31744a99fbc56edcc6af00bce4b989872b9c796331935ab6e3208bd6f48: Status 404 returned error can't find the container with id f231b31744a99fbc56edcc6af00bce4b989872b9c796331935ab6e3208bd6f48 Jul 21 21:01:35.365410 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.365306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:35.365546 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:35.365472 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:01:35.365546 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:35.365491 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c87df747f-8jzjp: secret "image-registry-tls" not found Jul 21 21:01:35.365546 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:35.365546 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls podName:a4a7e50e-67ca-4842-a470-e80103746333 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:37.365528619 +0000 UTC m=+74.310149675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls") pod "image-registry-c87df747f-8jzjp" (UID: "a4a7e50e-67ca-4842-a470-e80103746333") : secret "image-registry-tls" not found Jul 21 21:01:35.858651 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.858613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" event={"ID":"91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f","Type":"ContainerStarted","Data":"22541eb6fe2a0247746f968337928d4531b2cc1e27d8e95f56246ec1730c2c0b"} Jul 21 21:01:35.861486 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.861458 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" event={"ID":"43b06cae-7bf6-4450-b3ed-8b1384087e66","Type":"ContainerStarted","Data":"7adbb2d985ae0361bdf4616b843f72da241db94c4aa717a7fe7726b12505a7c6"} Jul 21 21:01:35.861613 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:35.861506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" event={"ID":"43b06cae-7bf6-4450-b3ed-8b1384087e66","Type":"ContainerStarted","Data":"f231b31744a99fbc56edcc6af00bce4b989872b9c796331935ab6e3208bd6f48"} Jul 21 21:01:36.866720 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:36.866690 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" event={"ID":"104d8201-c4ba-4d11-b47a-1db1a035dda7","Type":"ContainerStarted","Data":"e8ab1b6357dfb3334a9a2bd6edbcf3053decdda70593259a7bd4cdfb81c0584b"} Jul 21 21:01:36.868200 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:36.868164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" event={"ID":"91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f","Type":"ContainerStarted","Data":"9651e88056c4aa5769617a954ed822c630267128e30cb58c8d3600437f13bc92"} Jul 21 21:01:36.885221 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:36.885167 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" podStartSLOduration=1.881429452 podStartE2EDuration="3.885148862s" totalImagesPullingTime="2.00371941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:36.884255509 +0000 UTC m=+73.828876577" watchObservedRunningTime="2026-07-21 21:01:36.885148862 +0000 UTC m=+73.829769928" Jul 21 21:01:36.885428 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:36.885399 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-q9zsg" podStartSLOduration=3.885389415 podStartE2EDuration="3.885389415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:35.877996688 +0000 UTC m=+72.822617754" watchObservedRunningTime="2026-07-21 21:01:36.885389415 +0000 UTC m=+73.830010478" Jul 21 21:01:36.899963 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:36.899919 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j8hjz" podStartSLOduration=2.797887634 podStartE2EDuration="3.899907093s" totalImagesPullingTime="1.102019459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:36.899638302 +0000 UTC m=+73.844259367" watchObservedRunningTime="2026-07-21 21:01:36.899907093 +0000 UTC m=+73.844528183" Jul 21 21:01:36.978411 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:36.978348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:37.381690 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:37.381653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:37.381866 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:37.381764 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:01:37.381866 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:37.381776 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c87df747f-8jzjp: secret "image-registry-tls" not found Jul 21 21:01:37.381866 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:37.381831 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls podName:a4a7e50e-67ca-4842-a470-e80103746333 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:41.381818219 +0000 UTC m=+78.326439262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls") pod "image-registry-c87df747f-8jzjp" (UID: "a4a7e50e-67ca-4842-a470-e80103746333") : secret "image-registry-tls" not found Jul 21 21:01:38.031089 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.031058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:38.108107 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.108077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:38.403959 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.403883 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:38.434859 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.434831 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck"] Jul 21 21:01:38.436674 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.436655 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" Jul 21 21:01:38.440076 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.440051 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 21 21:01:38.441317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.441298 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 21 21:01:38.441405 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.441328 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bbdzf\"" Jul 21 21:01:38.459587 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.459559 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck"] Jul 21 21:01:38.490498 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.490468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6kg\" (UniqueName: \"kubernetes.io/projected/60eb76a3-006a-44bc-814e-e5ed1fc6f42a-kube-api-access-sn6kg\") pod \"migrator-7cc9c7ffd7-mscck\" (UID: \"60eb76a3-006a-44bc-814e-e5ed1fc6f42a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" Jul 21 21:01:38.506156 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.506124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:38.515107 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.515081 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tvcnx"] Jul 21 21:01:38.517126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.517108 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.520061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.520044 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vj8lm\"" Jul 21 21:01:38.520465 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.520436 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 21:01:38.520882 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.520682 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 21:01:38.533503 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.533478 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvcnx"] Jul 21 21:01:38.591349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.591324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.591480 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.591353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-crio-socket\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.591480 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.591443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-data-volume\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.591480 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.591470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvww\" (UniqueName: \"kubernetes.io/projected/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-api-access-nwvww\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.591576 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.591491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.591613 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.591589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6kg\" (UniqueName: \"kubernetes.io/projected/60eb76a3-006a-44bc-814e-e5ed1fc6f42a-kube-api-access-sn6kg\") pod \"migrator-7cc9c7ffd7-mscck\" (UID: \"60eb76a3-006a-44bc-814e-e5ed1fc6f42a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" Jul 21 21:01:38.594672 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.594649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:38.604611 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.604586 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6kg\" (UniqueName: \"kubernetes.io/projected/60eb76a3-006a-44bc-814e-e5ed1fc6f42a-kube-api-access-sn6kg\") pod \"migrator-7cc9c7ffd7-mscck\" (UID: \"60eb76a3-006a-44bc-814e-e5ed1fc6f42a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" Jul 21 21:01:38.692356 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.692491 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-crio-socket\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.692491 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692430 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-data-volume\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.692491 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvww\" (UniqueName: \"kubernetes.io/projected/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-api-access-nwvww\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.692491 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:38.692481 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:38.692491 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692486 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.692653 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692529 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-crio-socket\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.692653 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:38.692545 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:38.692653 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:38.692552 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:39.192533444 +0000 UTC m=+76.137154504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:38.692653 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:38.692594 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:39.192583362 +0000 UTC m=+76.137204436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : configmap "kube-rbac-proxy" not found Jul 21 21:01:38.692827 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.692809 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-data-volume\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.704916 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.704890 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvww\" (UniqueName: \"kubernetes.io/projected/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-api-access-nwvww\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:38.745655 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.745632 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" Jul 21 21:01:38.820034 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.819985 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:38.865891 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.865868 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck"] Jul 21 21:01:38.868244 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:38.868220 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60eb76a3_006a_44bc_814e_e5ed1fc6f42a.slice/crio-3525515846fc298fdf74a3e07795428915b2312d27e8e41da5c0d4ab53b29c08 WatchSource:0}: Error finding container 3525515846fc298fdf74a3e07795428915b2312d27e8e41da5c0d4ab53b29c08: Status 404 returned error can't find the container with id 3525515846fc298fdf74a3e07795428915b2312d27e8e41da5c0d4ab53b29c08 Jul 21 21:01:38.875336 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.875310 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" event={"ID":"60eb76a3-006a-44bc-814e-e5ed1fc6f42a","Type":"ContainerStarted","Data":"3525515846fc298fdf74a3e07795428915b2312d27e8e41da5c0d4ab53b29c08"} Jul 21 21:01:38.926504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:38.926473 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:39.197516 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:39.197480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:39.197946 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:39.197561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:39.197946 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:39.197630 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:39.197946 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:39.197656 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:39.197946 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:39.197717 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:40.197694917 +0000 UTC m=+77.142315961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : configmap "kube-rbac-proxy" not found Jul 21 21:01:39.197946 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:39.197738 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:40.197729042 +0000 UTC m=+77.142350106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:40.204724 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.204688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:40.205094 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.204749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:40.205094 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:40.204844 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:40.205094 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:40.204863 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:40.205094 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:40.204915 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:42.204901232 +0000 UTC m=+79.149522276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:40.205094 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:40.204930 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:42.204923818 +0000 UTC m=+79.149544860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : configmap "kube-rbac-proxy" not found Jul 21 21:01:40.412799 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.412715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:40.519593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.519564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:40.882790 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.882754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" event={"ID":"60eb76a3-006a-44bc-814e-e5ed1fc6f42a","Type":"ContainerStarted","Data":"4871aa1cf114e82ef5d371e8b5453a29f6a77956d3cb5b057525a6019e2f94b4"} Jul 21 21:01:40.882960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.882845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" event={"ID":"60eb76a3-006a-44bc-814e-e5ed1fc6f42a","Type":"ContainerStarted","Data":"41fba551f51bb06368d6dedfd8974687339c9463722066dfedcb4fba79491237"} Jul 21 21:01:40.911090 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:40.911039 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-mscck" podStartSLOduration=1.757920751 podStartE2EDuration="2.911026184s" totalImagesPullingTime="1.153105433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:40.909478234 +0000 UTC m=+77.854099310" watchObservedRunningTime="2026-07-21 21:01:40.911026184 +0000 UTC m=+77.855647248" Jul 21 21:01:41.115575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:41.115543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:41.255844 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:41.255816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:41.413468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:41.413429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:41.413652 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:41.413583 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:01:41.413652 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:41.413601 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c87df747f-8jzjp: secret "image-registry-tls" not found Jul 21 21:01:41.413743 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:41.413656 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls podName:a4a7e50e-67ca-4842-a470-e80103746333 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:49.413641367 +0000 UTC m=+86.358262409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls") pod "image-registry-c87df747f-8jzjp" (UID: "a4a7e50e-67ca-4842-a470-e80103746333") : secret "image-registry-tls" not found Jul 21 21:01:41.514232 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:41.514160 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wk9cv_3e4e3cde-b184-49f6-bb67-1d5962d17492/dns-node-resolver/0.log" Jul 21 21:01:41.770348 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:41.770271 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:42.217480 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:42.217441 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:42.217634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:42.217504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:42.217634 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:42.217593 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:42.217732 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:42.217657 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:46.217642215 +0000 UTC m=+83.162263258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:42.217732 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:42.217596 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:42.217732 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:42.217725 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:46.217714682 +0000 UTC m=+83.162335725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : configmap "kube-rbac-proxy" not found Jul 21 21:01:42.317326 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:42.317299 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7cgd8_8f71e36d-9129-487a-83bf-05ce9934d058/node-ca/0.log" Jul 21 21:01:42.912764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:42.912732 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5qq9n_666656f7-3729-44fa-a480-554e40f6e25c/serve-healthcheck-canary/0.log" Jul 21 21:01:43.513718 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:43.513691 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-mscck_60eb76a3-006a-44bc-814e-e5ed1fc6f42a/migrator/0.log" Jul 21 21:01:43.613772 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:43.613736 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:43.713330 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:43.713299 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-mscck_60eb76a3-006a-44bc-814e-e5ed1fc6f42a/graceful-termination/0.log" Jul 21 21:01:43.828314 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:43.828236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:43.923679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:43.923639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:44.049653 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:44.049620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:44.156898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:44.156811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:44.290163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:44.290131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:44.413069 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:44.413042 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:44.926472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:44.926436 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:45.527516 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:45.527481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:45.705460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:45.705430 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:46.118550 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:46.118519 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:46.248120 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:46.248076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:46.248290 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:46.248147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:46.248290 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:46.248232 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:46.248290 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:46.248244 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:46.248290 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:46.248291 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:54.248278639 +0000 UTC m=+91.192899685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : configmap "kube-rbac-proxy" not found Jul 21 21:01:46.248442 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:01:46.248304 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls podName:f2b3f4b5-8578-4775-8ac9-b8cedd981d32 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:54.248298102 +0000 UTC m=+91.192919144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvcnx" (UID: "f2b3f4b5-8578-4775-8ac9-b8cedd981d32") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:46.317871 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:46.317839 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:46.523601 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:46.523568 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:46.721526 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:46.721488 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:47.322694 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:47.322660 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:47.561407 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:47.561376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:47.911920 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:47.911884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:48.111033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:48.110999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:48.309640 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:48.309606 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:48.365950 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:48.365923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:48.609177 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:48.609091 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:48.718577 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:48.718541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:49.474635 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.474587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:49.477113 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.477094 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"image-registry-c87df747f-8jzjp\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:49.656935 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.656878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:49.721239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.721209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:49.785125 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.785068 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c87df747f-8jzjp"] Jul 21 21:01:49.787778 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:49.787752 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a7e50e_67ca_4842_a470_e80103746333.slice/crio-4e4c469ad410ce352614d8533d0be07f46fc36c0e89f0659526f8a311f1e4e5e WatchSource:0}: Error finding container 4e4c469ad410ce352614d8533d0be07f46fc36c0e89f0659526f8a311f1e4e5e: Status 404 returned error can't find the container with id 4e4c469ad410ce352614d8533d0be07f46fc36c0e89f0659526f8a311f1e4e5e Jul 21 21:01:49.910472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.910438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" event={"ID":"a4a7e50e-67ca-4842-a470-e80103746333","Type":"ContainerStarted","Data":"6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56"} Jul 21 21:01:49.910626 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.910485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" event={"ID":"a4a7e50e-67ca-4842-a470-e80103746333","Type":"ContainerStarted","Data":"4e4c469ad410ce352614d8533d0be07f46fc36c0e89f0659526f8a311f1e4e5e"} Jul 21 21:01:49.910626 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.910543 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:01:49.913171 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.913151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:49.931798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:49.931735 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" podStartSLOduration=16.931716404 podStartE2EDuration="16.931716404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:49.931007161 +0000 UTC m=+86.875628225" watchObservedRunningTime="2026-07-21 21:01:49.931716404 +0000 UTC m=+86.876337470" Jul 21 21:01:50.016736 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:50.016657 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:50.314808 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:50.314724 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:50.522417 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:50.522387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:51.117764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:51.117732 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:51.510149 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:51.510114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:51.610906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:51.610871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:51.914732 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:51.914696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:52.126891 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:52.126856 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:54.154496 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.154462 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:54.228923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.228884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:54.315852 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.315809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:54.316014 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.315878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:54.316474 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.316447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:54.318224 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.318198 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2b3f4b5-8578-4775-8ac9-b8cedd981d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvcnx\" (UID: \"f2b3f4b5-8578-4775-8ac9-b8cedd981d32\") " pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:54.324934 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.324910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:54.428339 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.428316 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvcnx" Jul 21 21:01:54.557910 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.557881 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvcnx"] Jul 21 21:01:54.560771 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:01:54.560734 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b3f4b5_8578_4775_8ac9_b8cedd981d32.slice/crio-027b8d67ae21870dac751fde94314c4f3efdd145d597fda7b86cdb36bcefd7ca WatchSource:0}: Error finding container 027b8d67ae21870dac751fde94314c4f3efdd145d597fda7b86cdb36bcefd7ca: Status 404 returned error can't find the container with id 027b8d67ae21870dac751fde94314c4f3efdd145d597fda7b86cdb36bcefd7ca Jul 21 21:01:54.631332 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.631305 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:01:54.929753 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:54.929717 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvcnx" event={"ID":"f2b3f4b5-8578-4775-8ac9-b8cedd981d32","Type":"ContainerStarted","Data":"027b8d67ae21870dac751fde94314c4f3efdd145d597fda7b86cdb36bcefd7ca"} Jul 21 21:01:59.952893 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:01:59.952858 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvcnx" event={"ID":"f2b3f4b5-8578-4775-8ac9-b8cedd981d32","Type":"ContainerStarted","Data":"5acca4a18bffebc7dec65717f156ae551bcab08a741454e6fa5e4a95e51a1f74"} Jul 21 21:02:00.259275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:00.259189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:00.383259 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:00.383225 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:00.460608 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:00.460575 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:00.567395 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:00.567298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:00.807243 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:00.807204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:00.928319 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:00.928288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:01.072925 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:01.072873 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:01.361347 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:01.361264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:01.418798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:01.418760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:01.592028 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:01.591995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:01.683060 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:01.683027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:01.946414 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:01.946320 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:02.149066 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:02.149033 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:02.324322 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:02.324250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:02.784937 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:02.784892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:02:02.787446 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:02.787415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe43a134-8bcb-41ac-8b93-f9d7f657f094-metrics-tls\") pod \"dns-default-rk9dq\" (UID: \"fe43a134-8bcb-41ac-8b93-f9d7f657f094\") " pod="openshift-dns/dns-default-rk9dq" Jul 21 21:02:02.834099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:02.834069 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wp8nm\"" Jul 21 21:02:02.841540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:02.841506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rk9dq" Jul 21 21:02:03.158590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.158521 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk"] Jul 21 21:02:03.160604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.160590 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.164598 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.164560 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 21:02:03.164598 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.164595 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 21:02:03.164847 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.164630 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 21:02:03.164847 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.164682 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 21:02:03.178348 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.178323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk"] Jul 21 21:02:03.187745 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.187718 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c87df747f-8jzjp"] Jul 21 21:02:03.219555 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.219529 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85857f96-zd47z"] Jul 21 21:02:03.221541 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.221519 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.244605 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.244573 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85857f96-zd47z"] Jul 21 21:02:03.288776 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2sms\" (UniqueName: \"kubernetes.io/projected/41b8f92e-5c74-4f04-823e-13fe2b1febd4-kube-api-access-t2sms\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.288907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288789 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6a4a246-9f16-4f17-98a6-21ea631ca67a-trusted-ca\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.288907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b8f92e-5c74-4f04-823e-13fe2b1febd4-tmp\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.288907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288858 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6a4a246-9f16-4f17-98a6-21ea631ca67a-registry-certificates\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.288907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288875 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-bound-sa-token\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.288907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6a4a246-9f16-4f17-98a6-21ea631ca67a-image-registry-private-configuration\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.289072 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncrz\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-kube-api-access-jncrz\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.289072 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.288990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6a4a246-9f16-4f17-98a6-21ea631ca67a-ca-trust-extracted\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.289072 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.289018 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6a4a246-9f16-4f17-98a6-21ea631ca67a-installation-pull-secrets\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.289072 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.289054 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-registry-tls\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.289189 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.289110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/41b8f92e-5c74-4f04-823e-13fe2b1febd4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.351598 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.351567 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rk9dq"] Jul 21 21:02:03.354095 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:03.354071 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe43a134_8bcb_41ac_8b93_f9d7f657f094.slice/crio-234d42b9be263b7abe027c05c0f3a016712c5486eb8d6f34827f7337742e5cdc WatchSource:0}: Error finding container 234d42b9be263b7abe027c05c0f3a016712c5486eb8d6f34827f7337742e5cdc: Status 404 returned error can't find the container with id 234d42b9be263b7abe027c05c0f3a016712c5486eb8d6f34827f7337742e5cdc Jul 21 21:02:03.390029 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/41b8f92e-5c74-4f04-823e-13fe2b1febd4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.390158 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2sms\" (UniqueName: \"kubernetes.io/projected/41b8f92e-5c74-4f04-823e-13fe2b1febd4-kube-api-access-t2sms\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.390158 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6a4a246-9f16-4f17-98a6-21ea631ca67a-trusted-ca\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390158 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b8f92e-5c74-4f04-823e-13fe2b1febd4-tmp\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.390158 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390112 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6a4a246-9f16-4f17-98a6-21ea631ca67a-registry-certificates\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390158 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-bound-sa-token\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390158 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390150 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6a4a246-9f16-4f17-98a6-21ea631ca67a-image-registry-private-configuration\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390488 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncrz\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-kube-api-access-jncrz\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390488 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6a4a246-9f16-4f17-98a6-21ea631ca67a-ca-trust-extracted\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390592 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6a4a246-9f16-4f17-98a6-21ea631ca67a-installation-pull-secrets\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390592 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-registry-tls\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.390688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.390621 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b8f92e-5c74-4f04-823e-13fe2b1febd4-tmp\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.391064 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.391001 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f6a4a246-9f16-4f17-98a6-21ea631ca67a-trusted-ca\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.391064 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.391042 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f6a4a246-9f16-4f17-98a6-21ea631ca67a-registry-certificates\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.391173 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.391135 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f6a4a246-9f16-4f17-98a6-21ea631ca67a-ca-trust-extracted\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.392830 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.392812 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/41b8f92e-5c74-4f04-823e-13fe2b1febd4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.393161 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.393142 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-registry-tls\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.393548 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.393524 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f6a4a246-9f16-4f17-98a6-21ea631ca67a-installation-pull-secrets\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.393627 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.393608 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f6a4a246-9f16-4f17-98a6-21ea631ca67a-image-registry-private-configuration\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.406483 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.406458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2sms\" (UniqueName: \"kubernetes.io/projected/41b8f92e-5c74-4f04-823e-13fe2b1febd4-kube-api-access-t2sms\") pod \"klusterlet-addon-workmgr-6fc77dc87-6bbpk\" (UID: \"41b8f92e-5c74-4f04-823e-13fe2b1febd4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.406888 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.406864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncrz\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-kube-api-access-jncrz\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.406984 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.406966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f6a4a246-9f16-4f17-98a6-21ea631ca67a-bound-sa-token\") pod \"image-registry-85857f96-zd47z\" (UID: \"f6a4a246-9f16-4f17-98a6-21ea631ca67a\") " pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.471507 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.471478 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:03.537110 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.536444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.601653 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.601622 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk"] Jul 21 21:02:03.606010 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:03.605978 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b8f92e_5c74_4f04_823e_13fe2b1febd4.slice/crio-b0306cd30f3c3d5370d3b49c306148fc3ba476ca8c89041378c697902e1f94f4 WatchSource:0}: Error finding container b0306cd30f3c3d5370d3b49c306148fc3ba476ca8c89041378c697902e1f94f4: Status 404 returned error can't find the container with id b0306cd30f3c3d5370d3b49c306148fc3ba476ca8c89041378c697902e1f94f4 Jul 21 21:02:03.666904 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.666835 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85857f96-zd47z"] Jul 21 21:02:03.668863 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:03.668838 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a4a246_9f16_4f17_98a6_21ea631ca67a.slice/crio-219c902e04712fefff06f373b069c5c8ce3298109408123571a26e8071c1ce97 WatchSource:0}: Error finding container 219c902e04712fefff06f373b069c5c8ce3298109408123571a26e8071c1ce97: Status 404 returned error can't find the container with id 219c902e04712fefff06f373b069c5c8ce3298109408123571a26e8071c1ce97 Jul 21 21:02:03.968061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.968019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rk9dq" event={"ID":"fe43a134-8bcb-41ac-8b93-f9d7f657f094","Type":"ContainerStarted","Data":"234d42b9be263b7abe027c05c0f3a016712c5486eb8d6f34827f7337742e5cdc"} Jul 21 21:02:03.972338 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.972272 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvcnx" event={"ID":"f2b3f4b5-8578-4775-8ac9-b8cedd981d32","Type":"ContainerStarted","Data":"465919ef6df8eb7805398e277cc4940df535fe27f6b239b78efb8e25dd969744"} Jul 21 21:02:03.974764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.974205 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85857f96-zd47z" event={"ID":"f6a4a246-9f16-4f17-98a6-21ea631ca67a","Type":"ContainerStarted","Data":"95dd1510759ff5bd9674e7e5316b6b04b374ff8cdcaaddceef6e361dfd941481"} Jul 21 21:02:03.974764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.974251 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85857f96-zd47z" event={"ID":"f6a4a246-9f16-4f17-98a6-21ea631ca67a","Type":"ContainerStarted","Data":"219c902e04712fefff06f373b069c5c8ce3298109408123571a26e8071c1ce97"} Jul 21 21:02:03.974764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.974339 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:03.976473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.976448 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" event={"ID":"41b8f92e-5c74-4f04-823e-13fe2b1febd4","Type":"ContainerStarted","Data":"b0306cd30f3c3d5370d3b49c306148fc3ba476ca8c89041378c697902e1f94f4"} Jul 21 21:02:03.995688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:03.995634 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85857f96-zd47z" podStartSLOduration=0.995620092 podStartE2EDuration="995.620092ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:03.994878964 +0000 UTC m=+100.939500044" watchObservedRunningTime="2026-07-21 21:02:03.995620092 +0000 UTC m=+100.940241156" Jul 21 21:02:05.003099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:05.003040 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:05.059986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:05.059915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:05.864794 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:05.864757 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lpdk5" Jul 21 21:02:06.350266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:06.350236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:06.450893 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:06.450862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:07.280527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:07.280483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:07.381147 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:07.381114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:10.354686 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:10.354625 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:10.452222 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:10.452191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:10.540002 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:10.539969 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:10.640845 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:10.640815 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:11.002524 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.002485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rk9dq" event={"ID":"fe43a134-8bcb-41ac-8b93-f9d7f657f094","Type":"ContainerStarted","Data":"812bdf6d70f9afe0ddc8d5d6a9c26fe5544a10e8ebe16b54e22980b8fc8a6a9a"} Jul 21 21:02:11.002700 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.002549 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rk9dq" Jul 21 21:02:11.002700 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.002575 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rk9dq" event={"ID":"fe43a134-8bcb-41ac-8b93-f9d7f657f094","Type":"ContainerStarted","Data":"8703c045c545ac6a2a5063a41f5bd9092cc3f2835bca2194e8f45d08bc888aa4"} Jul 21 21:02:11.004477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.004453 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvcnx" event={"ID":"f2b3f4b5-8578-4775-8ac9-b8cedd981d32","Type":"ContainerStarted","Data":"b29df34c5aa4740e3938be68663d9f54b1f9a78dd8c19358d51701042ba852d6"} Jul 21 21:02:11.005740 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.005715 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" event={"ID":"41b8f92e-5c74-4f04-823e-13fe2b1febd4","Type":"ContainerStarted","Data":"deec7dc8f4a59d44a0a1ced743e6e7935730a0192c452f9399248c2e9ea7c3f6"} Jul 21 21:02:11.005974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.005959 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:11.007573 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.007554 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" Jul 21 21:02:11.021168 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.021124 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rk9dq" podStartSLOduration=66.370749989 podStartE2EDuration="1m13.021111624s" totalImagesPullingTime="6.650361635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:11.020212006 +0000 UTC m=+107.964833072" watchObservedRunningTime="2026-07-21 21:02:11.021111624 +0000 UTC m=+107.965732687" Jul 21 21:02:11.041153 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.041114 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tvcnx" podStartSLOduration=17.705292214 podStartE2EDuration="33.041101681s" totalImagesPullingTime="15.335809467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:01:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:11.040800868 +0000 UTC m=+107.985421950" watchObservedRunningTime="2026-07-21 21:02:11.041101681 +0000 UTC m=+107.985722787" Jul 21 21:02:11.057726 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:11.057688 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6fc77dc87-6bbpk" podStartSLOduration=1.6578613679999998 podStartE2EDuration="8.057677322s" totalImagesPullingTime="6.399815954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:11.057242819 +0000 UTC m=+108.001863902" watchObservedRunningTime="2026-07-21 21:02:11.057677322 +0000 UTC m=+108.002298387" Jul 21 21:02:13.174759 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:13.174721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:13.193585 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:13.193560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:02:13.276307 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:13.276276 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:14.559857 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:14.559822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:14.634966 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:14.634936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:14.833871 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:14.833790 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:14.935244 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:14.935217 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:14.993039 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:14.993010 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:15.564284 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.564255 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78994ff884-mvkv6"] Jul 21 21:02:15.566350 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.566333 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.569301 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.569275 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 21:02:15.569301 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.569296 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 21:02:15.569499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.569275 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fz5rw\"" Jul 21 21:02:15.570413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.570346 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 21:02:15.570522 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.570446 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 21:02:15.570522 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.570465 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 21:02:15.570817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.570801 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 21:02:15.571850 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.571831 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 21:02:15.575446 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.575426 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 21:02:15.577487 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.577466 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78994ff884-mvkv6"] Jul 21 21:02:15.689706 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-config\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.689880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-trusted-ca-bundle\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.689880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-service-ca\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.689880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-oauth-config\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.689880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689807 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-serving-cert\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.689880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-oauth-serving-cert\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.689880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.689853 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8l9\" (UniqueName: \"kubernetes.io/projected/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-kube-api-access-gr8l9\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.790984 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.790937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-config\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.790984 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.790990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-trusted-ca-bundle\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.791161 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-service-ca\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.791161 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-oauth-config\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.791241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-serving-cert\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.791241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-oauth-serving-cert\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.791338 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8l9\" (UniqueName: \"kubernetes.io/projected/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-kube-api-access-gr8l9\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.791895 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-service-ca\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.792025 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791872 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-config\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.792025 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.791955 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-oauth-serving-cert\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.792025 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.792013 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-trusted-ca-bundle\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.793797 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.793773 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-serving-cert\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.793908 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.793882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-oauth-config\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.800315 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.800292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8l9\" (UniqueName: \"kubernetes.io/projected/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-kube-api-access-gr8l9\") pod \"console-78994ff884-mvkv6\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:15.876726 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:15.876667 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:16.002525 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:16.002502 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78994ff884-mvkv6"] Jul 21 21:02:16.004523 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:16.004497 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5bc631_a250_4f94_b3b3_218bb0e8f2b3.slice/crio-3525d4dae88b6249baee546de4a6401663f1ac7d9110a3f87f414e96dcf552da WatchSource:0}: Error finding container 3525d4dae88b6249baee546de4a6401663f1ac7d9110a3f87f414e96dcf552da: Status 404 returned error can't find the container with id 3525d4dae88b6249baee546de4a6401663f1ac7d9110a3f87f414e96dcf552da Jul 21 21:02:16.024037 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:16.024009 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78994ff884-mvkv6" event={"ID":"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3","Type":"ContainerStarted","Data":"3525d4dae88b6249baee546de4a6401663f1ac7d9110a3f87f414e96dcf552da"} Jul 21 21:02:16.358237 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:16.358202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:16.460471 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:16.460423 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:19.037148 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:19.037064 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78994ff884-mvkv6" event={"ID":"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3","Type":"ContainerStarted","Data":"d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d"} Jul 21 21:02:19.060213 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:19.060159 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78994ff884-mvkv6" podStartSLOduration=1.394404152 podStartE2EDuration="4.060142723s" totalImagesPullingTime="2.665738571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:19.059854937 +0000 UTC m=+116.004476014" watchObservedRunningTime="2026-07-21 21:02:19.060142723 +0000 UTC m=+116.004763787" Jul 21 21:02:21.011214 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:21.011178 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rk9dq" Jul 21 21:02:24.376805 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:24.376739 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:24.477186 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:24.477146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:24.985559 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:24.985522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85857f96-zd47z" Jul 21 21:02:25.022554 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.022507 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:25.127889 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.127852 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:25.195075 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.195037 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:25.257540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.257467 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:25.877267 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.877237 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:25.877741 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.877294 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:25.882117 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:25.882094 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:26.065457 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.065420 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:02:26.224509 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.224480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:26.276660 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.276636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:26.695943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.695911 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:26.959999 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.959923 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4"] Jul 21 21:02:26.962761 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.962745 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:26.969386 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.969346 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 21:02:26.969536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.969497 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 21:02:26.969536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.969498 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cvzf2\"" Jul 21 21:02:26.969704 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.969619 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 21:02:26.973754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.973737 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 21:02:26.973862 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.973840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:26.973908 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.973889 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 21:02:26.973908 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.973900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8sk\" (UniqueName: \"kubernetes.io/projected/ad54dbda-7a2f-44a8-9063-9d017a5617f9-kube-api-access-8w8sk\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:26.973976 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.973921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:26.973976 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.973946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad54dbda-7a2f-44a8-9063-9d017a5617f9-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:26.996574 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:26.996548 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4"] Jul 21 21:02:27.006203 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.006180 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:27.073739 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.073713 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-78lvl"] Jul 21 21:02:27.074677 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.074651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad54dbda-7a2f-44a8-9063-9d017a5617f9-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.074798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.074776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.074895 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.074869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8sk\" (UniqueName: \"kubernetes.io/projected/ad54dbda-7a2f-44a8-9063-9d017a5617f9-kube-api-access-8w8sk\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.075048 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.074986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.075177 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:02:27.075153 2570 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 21 21:02:27.075276 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:02:27.075216 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-tls podName:ad54dbda-7a2f-44a8-9063-9d017a5617f9 nodeName:}" failed. No retries permitted until 2026-07-21 21:02:27.575201468 +0000 UTC m=+124.519822510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-ftzq4" (UID: "ad54dbda-7a2f-44a8-9063-9d017a5617f9") : secret "openshift-state-metrics-tls" not found Jul 21 21:02:27.075514 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.075498 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad54dbda-7a2f-44a8-9063-9d017a5617f9-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.076346 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.076326 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.077575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.077555 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.080222 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.080206 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6hrnm\"" Jul 21 21:02:27.081215 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.081201 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 21:02:27.081590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.081575 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 21:02:27.081648 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.081604 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 21:02:27.084923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.084902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8sk\" (UniqueName: \"kubernetes.io/projected/ad54dbda-7a2f-44a8-9063-9d017a5617f9-kube-api-access-8w8sk\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.137692 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.137663 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:27.175870 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.175843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-tls\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.175979 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.175883 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-metrics-client-ca\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.175979 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.175909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.175979 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.175959 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-sys\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.176097 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.176002 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkqj\" (UniqueName: \"kubernetes.io/projected/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-kube-api-access-stkqj\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.176097 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.176026 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-wtmp\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.176097 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.176080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-textfile\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.176205 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.176098 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-root\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.176205 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.176123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.235227 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.235177 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:27.276504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-tls\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276607 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-metrics-client-ca\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276672 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276714 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-sys\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276755 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276731 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkqj\" (UniqueName: \"kubernetes.io/projected/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-kube-api-access-stkqj\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276794 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276764 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-wtmp\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276844 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276827 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-sys\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276887 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-textfile\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276936 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-root\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.276995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.276971 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-wtmp\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.277094 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.277034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-root\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.277146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.277113 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-textfile\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.277197 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.277169 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-metrics-client-ca\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.277257 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.277194 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.279096 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.279075 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-tls\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.279457 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.279439 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.288510 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.288476 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkqj\" (UniqueName: \"kubernetes.io/projected/7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd-kube-api-access-stkqj\") pod \"node-exporter-78lvl\" (UID: \"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd\") " pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.336314 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.336284 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:27.395676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.395657 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78lvl" Jul 21 21:02:27.405083 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:27.405059 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d33f92a_28a6_4959_ac85_1a2ec6d3b9bd.slice/crio-2226b33b22764a9eac3e345ead62c1b4577a86e9c5946f115dbc14aae703cd03 WatchSource:0}: Error finding container 2226b33b22764a9eac3e345ead62c1b4577a86e9c5946f115dbc14aae703cd03: Status 404 returned error can't find the container with id 2226b33b22764a9eac3e345ead62c1b4577a86e9c5946f115dbc14aae703cd03 Jul 21 21:02:27.436131 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.436107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:27.536890 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.536789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:27.580742 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.580712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.583089 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.583062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54dbda-7a2f-44a8-9063-9d017a5617f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ftzq4\" (UID: \"ad54dbda-7a2f-44a8-9063-9d017a5617f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.872185 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.872110 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" Jul 21 21:02:27.949949 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:27.949915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:28.014149 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.014060 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4"] Jul 21 21:02:28.017975 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:28.017944 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad54dbda_7a2f_44a8_9063_9d017a5617f9.slice/crio-42135ca8540b47db4069b9e10a529a7f87acd34eda787b236b3a27142cd2e702 WatchSource:0}: Error finding container 42135ca8540b47db4069b9e10a529a7f87acd34eda787b236b3a27142cd2e702: Status 404 returned error can't find the container with id 42135ca8540b47db4069b9e10a529a7f87acd34eda787b236b3a27142cd2e702 Jul 21 21:02:28.038071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.038046 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:28.069566 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.069533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" event={"ID":"ad54dbda-7a2f-44a8-9063-9d017a5617f9","Type":"ContainerStarted","Data":"42135ca8540b47db4069b9e10a529a7f87acd34eda787b236b3a27142cd2e702"} Jul 21 21:02:28.070956 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.070927 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78lvl" event={"ID":"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd","Type":"ContainerStarted","Data":"2226b33b22764a9eac3e345ead62c1b4577a86e9c5946f115dbc14aae703cd03"} Jul 21 21:02:28.210923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.210874 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" podUID="a4a7e50e-67ca-4842-a470-e80103746333" containerName="registry" containerID="cri-o://6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56" gracePeriod=30 Jul 21 21:02:28.351271 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.351235 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:28.451612 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.451582 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:28.493294 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.493266 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:02:28.589842 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.589803 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-registry-certificates\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.589993 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.589845 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-bound-sa-token\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.589993 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.589875 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-image-registry-private-configuration\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.589993 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.589904 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2wz4\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-kube-api-access-g2wz4\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.589993 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.589934 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.590191 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.590004 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-installation-pull-secrets\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.590191 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.590052 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-trusted-ca\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.590191 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.590078 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4a7e50e-67ca-4842-a470-e80103746333-ca-trust-extracted\") pod \"a4a7e50e-67ca-4842-a470-e80103746333\" (UID: \"a4a7e50e-67ca-4842-a470-e80103746333\") " Jul 21 21:02:28.590324 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.590239 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:02:28.590516 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.590485 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:02:28.592516 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.592460 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:02:28.592611 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.592585 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:02:28.592704 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.592679 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:02:28.592763 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.592679 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-kube-api-access-g2wz4" (OuterVolumeSpecName: "kube-api-access-g2wz4") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "kube-api-access-g2wz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:02:28.592829 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.592810 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:02:28.598448 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.598429 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a7e50e-67ca-4842-a470-e80103746333-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a4a7e50e-67ca-4842-a470-e80103746333" (UID: "a4a7e50e-67ca-4842-a470-e80103746333"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:02:28.691489 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691464 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-installation-pull-secrets\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691489 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691488 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-trusted-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691496 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4a7e50e-67ca-4842-a470-e80103746333-ca-trust-extracted\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691505 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4a7e50e-67ca-4842-a470-e80103746333-registry-certificates\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691513 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-bound-sa-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691522 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4a7e50e-67ca-4842-a470-e80103746333-image-registry-private-configuration\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691530 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wz4\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-kube-api-access-g2wz4\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:28.691637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:28.691537 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4a7e50e-67ca-4842-a470-e80103746333-registry-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:02:29.075765 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.075684 2570 generic.go:356] "Generic (PLEG): container finished" podID="a4a7e50e-67ca-4842-a470-e80103746333" containerID="6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56" exitCode=0 Jul 21 21:02:29.075765 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.075741 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" Jul 21 21:02:29.076240 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.075773 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" event={"ID":"a4a7e50e-67ca-4842-a470-e80103746333","Type":"ContainerDied","Data":"6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56"} Jul 21 21:02:29.076240 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.075841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c87df747f-8jzjp" event={"ID":"a4a7e50e-67ca-4842-a470-e80103746333","Type":"ContainerDied","Data":"4e4c469ad410ce352614d8533d0be07f46fc36c0e89f0659526f8a311f1e4e5e"} Jul 21 21:02:29.076240 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.075876 2570 scope.go:117] "RemoveContainer" containerID="6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56" Jul 21 21:02:29.077987 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.077848 2570 generic.go:356] "Generic (PLEG): container finished" podID="7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd" containerID="3c7bc7b6605df08d252fcfc5d5447aa482687493f622534b0bef37fe20df6b8e" exitCode=0 Jul 21 21:02:29.077987 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.077931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78lvl" event={"ID":"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd","Type":"ContainerDied","Data":"3c7bc7b6605df08d252fcfc5d5447aa482687493f622534b0bef37fe20df6b8e"} Jul 21 21:02:29.081185 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.081156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" event={"ID":"ad54dbda-7a2f-44a8-9063-9d017a5617f9","Type":"ContainerStarted","Data":"6d705465370a42f57558fcf9e38017598fa3d81011c79d68d868a61e28d055fe"} Jul 21 21:02:29.081294 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.081215 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" event={"ID":"ad54dbda-7a2f-44a8-9063-9d017a5617f9","Type":"ContainerStarted","Data":"8bb62f270915d00247583591e7f8f47557cd19ac23763f2208b0eb56a83da0c4"} Jul 21 21:02:29.114242 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.114217 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c87df747f-8jzjp"] Jul 21 21:02:29.117886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.117859 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c87df747f-8jzjp"] Jul 21 21:02:29.198887 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.198863 2570 scope.go:117] "RemoveContainer" containerID="6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56" Jul 21 21:02:29.199255 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:02:29.199226 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56\": container with ID starting with 6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56 not found: ID does not exist" containerID="6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56" Jul 21 21:02:29.199355 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.199263 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56"} err="failed to get container status \"6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56\": rpc error: code = NotFound desc = could not find container \"6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56\": container with ID starting with 6de066b664c526a39400f6629ac2236f4e192b5fe31ca8392c737787dd21fc56 not found: ID does not exist" Jul 21 21:02:29.457189 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.457159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:29.554930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.554901 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a7e50e-67ca-4842-a470-e80103746333" path="/var/lib/kubelet/pods/a4a7e50e-67ca-4842-a470-e80103746333/volumes" Jul 21 21:02:29.558439 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.558412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:29.657971 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.657942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:29.920796 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.920756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:29.964132 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.964104 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69c9fd97-tdkpc"] Jul 21 21:02:29.964403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.964390 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a7e50e-67ca-4842-a470-e80103746333" containerName="registry" Jul 21 21:02:29.964452 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.964406 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a7e50e-67ca-4842-a470-e80103746333" containerName="registry" Jul 21 21:02:29.964491 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.964465 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a7e50e-67ca-4842-a470-e80103746333" containerName="registry" Jul 21 21:02:29.966757 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.966742 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:29.969693 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.969677 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 21:02:29.970063 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.970037 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 21:02:29.970171 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.970112 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2lz7w\"" Jul 21 21:02:29.970929 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.970915 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 21:02:29.970929 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.970925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 21:02:29.971071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.970933 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ed32lbaqc9ufg\"" Jul 21 21:02:29.971071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.971002 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 21:02:29.983536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:29.983518 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c9fd97-tdkpc"] Jul 21 21:02:30.000530 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sz4\" (UniqueName: \"kubernetes.io/projected/04b6e8fa-b025-4c01-9c71-c347a772a61f-kube-api-access-d4sz4\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-tls\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04b6e8fa-b025-4c01-9c71-c347a772a61f-metrics-client-ca\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000637 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-grpc-tls\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000680 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.000747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.000696 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.016771 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.016727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:30.086199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.086169 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" event={"ID":"ad54dbda-7a2f-44a8-9063-9d017a5617f9","Type":"ContainerStarted","Data":"9d4ac3b8b2c7ca683e930ad1ac4227b4d27f77b317f0d443452a82bac9441f53"} Jul 21 21:02:30.088812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.088787 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78lvl" event={"ID":"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd","Type":"ContainerStarted","Data":"9af807cd966239696b5d1b06e5bb18ca56c60e1ba126c3982c6fc2d594d0b260"} Jul 21 21:02:30.088898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.088836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78lvl" event={"ID":"7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd","Type":"ContainerStarted","Data":"e45b8cf06fc44c1c0909f52e69553e27b7522c67eb892e012d06814f8bdc1859"} Jul 21 21:02:30.095532 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.095482 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:30.101609 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.101590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-grpc-tls\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.101657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.101622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.101657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.101653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.101836 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.101816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.101905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.101890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sz4\" (UniqueName: \"kubernetes.io/projected/04b6e8fa-b025-4c01-9c71-c347a772a61f-kube-api-access-d4sz4\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.102702 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.101986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-tls\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.102702 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.102013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04b6e8fa-b025-4c01-9c71-c347a772a61f-metrics-client-ca\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.102702 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.102053 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.103094 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.103071 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04b6e8fa-b025-4c01-9c71-c347a772a61f-metrics-client-ca\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.104464 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.104445 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-grpc-tls\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.104594 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.104575 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.104901 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.104883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.105027 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.105010 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.105388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.105350 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-tls\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.105454 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.105409 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04b6e8fa-b025-4c01-9c71-c347a772a61f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.106697 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.106658 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ftzq4" podStartSLOduration=3.044042021 podStartE2EDuration="4.106646315s" totalImagesPullingTime="1.062604294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:30.106032407 +0000 UTC m=+127.050653488" watchObservedRunningTime="2026-07-21 21:02:30.106646315 +0000 UTC m=+127.051267380" Jul 21 21:02:30.111300 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.111282 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sz4\" (UniqueName: \"kubernetes.io/projected/04b6e8fa-b025-4c01-9c71-c347a772a61f-kube-api-access-d4sz4\") pod \"thanos-querier-69c9fd97-tdkpc\" (UID: \"04b6e8fa-b025-4c01-9c71-c347a772a61f\") " pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.127056 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.127017 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-78lvl" podStartSLOduration=2.073626374 podStartE2EDuration="3.127005257s" totalImagesPullingTime="978.313676ms" totalInitContainerRuntime="75.065207ms" isStatefulPod=true podCreationTimestamp="2026-07-21 21:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:30.12667837 +0000 UTC m=+127.071299465" watchObservedRunningTime="2026-07-21 21:02:30.127005257 +0000 UTC m=+127.071626323" Jul 21 21:02:30.275423 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.275330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:30.403844 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.403822 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c9fd97-tdkpc"] Jul 21 21:02:30.406208 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:30.406183 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b6e8fa_b025_4c01_9c71_c347a772a61f.slice/crio-0a1fe8651738dfcefd90a703a4a083402c6b77dbc0f7b171179c2746d077f077 WatchSource:0}: Error finding container 0a1fe8651738dfcefd90a703a4a083402c6b77dbc0f7b171179c2746d077f077: Status 404 returned error can't find the container with id 0a1fe8651738dfcefd90a703a4a083402c6b77dbc0f7b171179c2746d077f077 Jul 21 21:02:30.514772 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:30.514734 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:31.095143 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.095093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"0a1fe8651738dfcefd90a703a4a083402c6b77dbc0f7b171179c2746d077f077"} Jul 21 21:02:31.347144 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.347039 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55f5b4445c-kp7vq"] Jul 21 21:02:31.349522 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.349500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.353287 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.353256 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 21:02:31.353287 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.353268 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 21:02:31.353287 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.353283 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 21:02:31.353499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.353316 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 21:02:31.353499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.353274 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xnz9r\"" Jul 21 21:02:31.353499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.353264 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7qjj21lg6dbi\"" Jul 21 21:02:31.361187 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.361167 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55f5b4445c-kp7vq"] Jul 21 21:02:31.411318 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-secret-metrics-server-client-certs\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.411473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411351 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49dl\" (UniqueName: \"kubernetes.io/projected/0160d1e1-ae4a-4a21-8b37-878ed30f763d-kube-api-access-p49dl\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.411538 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0160d1e1-ae4a-4a21-8b37-878ed30f763d-metrics-server-audit-profiles\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.411538 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-client-ca-bundle\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.411622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411545 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0160d1e1-ae4a-4a21-8b37-878ed30f763d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.411668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411616 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0160d1e1-ae4a-4a21-8b37-878ed30f763d-audit-log\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.411668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.411652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-secret-metrics-server-tls\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512480 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-client-ca-bundle\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512619 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0160d1e1-ae4a-4a21-8b37-878ed30f763d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512619 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512523 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0160d1e1-ae4a-4a21-8b37-878ed30f763d-audit-log\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512619 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-secret-metrics-server-tls\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512619 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-secret-metrics-server-client-certs\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512824 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49dl\" (UniqueName: \"kubernetes.io/projected/0160d1e1-ae4a-4a21-8b37-878ed30f763d-kube-api-access-p49dl\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.512824 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.512706 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0160d1e1-ae4a-4a21-8b37-878ed30f763d-metrics-server-audit-profiles\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.513072 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.513001 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0160d1e1-ae4a-4a21-8b37-878ed30f763d-audit-log\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.513982 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.513955 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0160d1e1-ae4a-4a21-8b37-878ed30f763d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.514152 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.514041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0160d1e1-ae4a-4a21-8b37-878ed30f763d-metrics-server-audit-profiles\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.515413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.515394 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-secret-metrics-server-client-certs\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.515593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.515573 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-client-ca-bundle\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.516150 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.516129 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0160d1e1-ae4a-4a21-8b37-878ed30f763d-secret-metrics-server-tls\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.522119 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.522094 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49dl\" (UniqueName: \"kubernetes.io/projected/0160d1e1-ae4a-4a21-8b37-878ed30f763d-kube-api-access-p49dl\") pod \"metrics-server-55f5b4445c-kp7vq\" (UID: \"0160d1e1-ae4a-4a21-8b37-878ed30f763d\") " pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.662596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.662513 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:31.736860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.736779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:31.738820 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.738797 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546475b69b-c5w5l"] Jul 21 21:02:31.742026 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.741993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.756036 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.755882 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546475b69b-c5w5l"] Jul 21 21:02:31.811657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.811624 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55f5b4445c-kp7vq"] Jul 21 21:02:31.814681 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.814654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-console-config\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.814806 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.814727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-trusted-ca-bundle\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.814863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.814799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-oauth-config\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.814916 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.814889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-serving-cert\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.814985 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.814945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlv6g\" (UniqueName: \"kubernetes.io/projected/04b0a036-96d4-4a8d-9e86-89e48db73617-kube-api-access-xlv6g\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.815042 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.815005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:31.815042 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.815005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-service-ca\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.815114 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.815054 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-oauth-serving-cert\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.915922 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.915846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-console-config\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.915936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-trusted-ca-bundle\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.915977 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-oauth-config\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-serving-cert\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlv6g\" (UniqueName: \"kubernetes.io/projected/04b0a036-96d4-4a8d-9e86-89e48db73617-kube-api-access-xlv6g\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916348 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-service-ca\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916348 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-oauth-serving-cert\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.916751 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-console-config\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.917155 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916814 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-oauth-serving-cert\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.917155 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916861 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-trusted-ca-bundle\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.917155 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.916957 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-service-ca\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.919045 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.919026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-serving-cert\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.919430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.919406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-oauth-config\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.925189 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.925168 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlv6g\" (UniqueName: \"kubernetes.io/projected/04b0a036-96d4-4a8d-9e86-89e48db73617-kube-api-access-xlv6g\") pod \"console-546475b69b-c5w5l\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:31.958022 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:31.957986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:32.037451 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:32.037419 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:32.052394 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:32.052346 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:32.312623 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:32.312548 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0160d1e1_ae4a_4a21_8b37_878ed30f763d.slice/crio-a983f089ccc65a497892609587de82ff593a634357cd6ca8b929c3c446f42590 WatchSource:0}: Error finding container a983f089ccc65a497892609587de82ff593a634357cd6ca8b929c3c446f42590: Status 404 returned error can't find the container with id a983f089ccc65a497892609587de82ff593a634357cd6ca8b929c3c446f42590 Jul 21 21:02:32.435175 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:32.435152 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546475b69b-c5w5l"] Jul 21 21:02:32.508322 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:32.508289 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b0a036_96d4_4a8d_9e86_89e48db73617.slice/crio-3cdbe6a4951ed22583edb49f00ef71481cbae2d3e03e9351d8bc8685e979c44b WatchSource:0}: Error finding container 3cdbe6a4951ed22583edb49f00ef71481cbae2d3e03e9351d8bc8685e979c44b: Status 404 returned error can't find the container with id 3cdbe6a4951ed22583edb49f00ef71481cbae2d3e03e9351d8bc8685e979c44b Jul 21 21:02:33.105767 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.105676 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"de7ecb57e478f1401cfdda9d7114449bf5f5856fb199f68ae1e92ed8c16e544d"} Jul 21 21:02:33.105909 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.105804 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"e24ca494701718d48a22f27988835c3014a858d2c75a56cd089bd8ad0dfd19dc"} Jul 21 21:02:33.105909 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.105848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"6debbd0aa8e10cf1ff15b0951ec3159c0c824658d408a03f95ac97b26a2f19f9"} Jul 21 21:02:33.107136 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.107108 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" event={"ID":"0160d1e1-ae4a-4a21-8b37-878ed30f763d","Type":"ContainerStarted","Data":"a983f089ccc65a497892609587de82ff593a634357cd6ca8b929c3c446f42590"} Jul 21 21:02:33.108722 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.108690 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546475b69b-c5w5l" event={"ID":"04b0a036-96d4-4a8d-9e86-89e48db73617","Type":"ContainerStarted","Data":"2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6"} Jul 21 21:02:33.108836 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.108735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546475b69b-c5w5l" event={"ID":"04b0a036-96d4-4a8d-9e86-89e48db73617","Type":"ContainerStarted","Data":"3cdbe6a4951ed22583edb49f00ef71481cbae2d3e03e9351d8bc8685e979c44b"} Jul 21 21:02:33.127250 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.127200 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546475b69b-c5w5l" podStartSLOduration=2.127181676 podStartE2EDuration="2.127181676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:33.12638564 +0000 UTC m=+130.071006725" watchObservedRunningTime="2026-07-21 21:02:33.127181676 +0000 UTC m=+130.071802741" Jul 21 21:02:33.171740 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.171706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:33.431778 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.431741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:02:33.434091 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.434074 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2742e7c0-42ac-4ed5-89dc-e77b9fb2381d-metrics-certs\") pod \"network-metrics-daemon-twf9r\" (UID: \"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d\") " pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:02:33.579927 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.579895 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvwws\"" Jul 21 21:02:33.587440 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:33.587408 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-twf9r" Jul 21 21:02:34.578100 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:34.578065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:34.680031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:34.680006 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:34.749222 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:34.749100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-twf9r"] Jul 21 21:02:34.751458 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:02:34.751431 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2742e7c0_42ac_4ed5_89dc_e77b9fb2381d.slice/crio-f5350f13bf4f1cdb677fd7b9cb80f11e05e1ff00c4aa10ffcb9f0182801296d0 WatchSource:0}: Error finding container f5350f13bf4f1cdb677fd7b9cb80f11e05e1ff00c4aa10ffcb9f0182801296d0: Status 404 returned error can't find the container with id f5350f13bf4f1cdb677fd7b9cb80f11e05e1ff00c4aa10ffcb9f0182801296d0 Jul 21 21:02:35.121667 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.121575 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"80701b127def649f3e9d4a8575dd18d5a29d710ef4ca0534fbada23db8727da7"} Jul 21 21:02:35.121815 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.121693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"9a3b166a86edf34214426e22810342a287c35a216c5b7ea0799fae1bd735485a"} Jul 21 21:02:35.121815 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.121739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" event={"ID":"04b6e8fa-b025-4c01-9c71-c347a772a61f","Type":"ContainerStarted","Data":"c4841f027c6d62fab132b7d24e8a1b868c2af15c7fc7c68a452e3ef3afcc8276"} Jul 21 21:02:35.121933 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.121903 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:35.122893 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.122863 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twf9r" event={"ID":"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d","Type":"ContainerStarted","Data":"f5350f13bf4f1cdb677fd7b9cb80f11e05e1ff00c4aa10ffcb9f0182801296d0"} Jul 21 21:02:35.124578 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.124532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" event={"ID":"0160d1e1-ae4a-4a21-8b37-878ed30f763d","Type":"ContainerStarted","Data":"77a4d2e8d59f250f876ca5cc8e3e6e0bcac0522a6dfc5ba22eff448d0ebe02aa"} Jul 21 21:02:35.147858 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.147816 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" podStartSLOduration=2.248564285 podStartE2EDuration="6.147800884s" totalImagesPullingTime="3.899236599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:35.145020298 +0000 UTC m=+132.089641381" watchObservedRunningTime="2026-07-21 21:02:35.147800884 +0000 UTC m=+132.092421967" Jul 21 21:02:35.163640 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:35.163595 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" podStartSLOduration=1.856074208 podStartE2EDuration="4.163578236s" totalImagesPullingTime="2.307504028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:35.161395516 +0000 UTC m=+132.106016583" watchObservedRunningTime="2026-07-21 21:02:35.163578236 +0000 UTC m=+132.108199304" Jul 21 21:02:36.130495 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:36.130460 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twf9r" event={"ID":"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d","Type":"ContainerStarted","Data":"ff341cc6badca69cef1d7de345db8213f19ebac545026f9094f5a9527b5c9bb9"} Jul 21 21:02:36.356459 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:36.356425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:36.457726 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:36.457691 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:37.136402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:37.136345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-twf9r" event={"ID":"2742e7c0-42ac-4ed5-89dc-e77b9fb2381d","Type":"ContainerStarted","Data":"74ff49382281381cbf0608496f3907372606af728abbb5cb229dbd33a6a05364"} Jul 21 21:02:37.160047 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:37.159995 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-twf9r" podStartSLOduration=131.969911836 podStartE2EDuration="2m13.159979226s" totalImagesPullingTime="1.19006739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:37.15860754 +0000 UTC m=+134.103228607" watchObservedRunningTime="2026-07-21 21:02:37.159979226 +0000 UTC m=+134.104600294" Jul 21 21:02:37.285733 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:37.285701 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:37.343549 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:37.343524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:38.150403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:38.150353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:38.235947 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:38.235909 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:38.537682 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:38.537634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:38.605978 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:38.605937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:41.137148 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:41.137119 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69c9fd97-tdkpc" Jul 21 21:02:42.053575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.053537 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:42.053793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.053611 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:42.058389 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.058347 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:42.158515 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.158480 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:02:42.210010 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.209982 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78994ff884-mvkv6"] Jul 21 21:02:42.351863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.351771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:42.423754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.423717 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:42.477926 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:42.477888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:48.177413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:48.177355 2570 generic.go:356] "Generic (PLEG): container finished" podID="104d8201-c4ba-4d11-b47a-1db1a035dda7" containerID="e8ab1b6357dfb3334a9a2bd6edbcf3053decdda70593259a7bd4cdfb81c0584b" exitCode=0 Jul 21 21:02:48.177798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:48.177429 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" event={"ID":"104d8201-c4ba-4d11-b47a-1db1a035dda7","Type":"ContainerDied","Data":"e8ab1b6357dfb3334a9a2bd6edbcf3053decdda70593259a7bd4cdfb81c0584b"} Jul 21 21:02:48.177798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:48.177792 2570 scope.go:117] "RemoveContainer" containerID="e8ab1b6357dfb3334a9a2bd6edbcf3053decdda70593259a7bd4cdfb81c0584b" Jul 21 21:02:49.184605 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:49.184568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2rwnh" event={"ID":"104d8201-c4ba-4d11-b47a-1db1a035dda7","Type":"ContainerStarted","Data":"f950f9eb53e8d96fb529596ff6917d3609a25973db2c6d4d2f0c85ddbb0dc21f"} Jul 21 21:02:51.348244 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:51.348215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:51.405167 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:51.405132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:51.663023 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:51.662939 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:51.663169 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:51.663018 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:02:54.469181 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:54.469144 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:02:54.533590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:02:54.533554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:06.357852 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:06.357807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:06.458149 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:06.458119 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:07.231576 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.231536 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78994ff884-mvkv6" podUID="8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" containerName="console" containerID="cri-o://d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d" gracePeriod=15 Jul 21 21:03:07.473944 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.473921 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:03:07.622617 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622546 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-oauth-config\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.622617 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622590 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-config\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.622617 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622613 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr8l9\" (UniqueName: \"kubernetes.io/projected/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-kube-api-access-gr8l9\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.622863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622640 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-serving-cert\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.622863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622668 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-oauth-serving-cert\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.622863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622691 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-trusted-ca-bundle\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.622863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.622714 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-service-ca\") pod \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\" (UID: \"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3\") " Jul 21 21:03:07.623058 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.623020 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-config" (OuterVolumeSpecName: "console-config") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:03:07.623125 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.623083 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:03:07.623177 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.623131 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-service-ca" (OuterVolumeSpecName: "service-ca") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:03:07.623177 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.623149 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:03:07.625005 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.624981 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:03:07.625113 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.625053 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:03:07.625113 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.625091 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-kube-api-access-gr8l9" (OuterVolumeSpecName: "kube-api-access-gr8l9") pod "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" (UID: "8e5bc631-a250-4f94-b3b3-218bb0e8f2b3"). InnerVolumeSpecName "kube-api-access-gr8l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:03:07.724101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724065 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-oauth-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:07.724101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724095 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:07.724275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724108 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr8l9\" (UniqueName: \"kubernetes.io/projected/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-kube-api-access-gr8l9\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:07.724275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724120 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-console-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:07.724275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724135 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-oauth-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:07.724275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724145 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-trusted-ca-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:07.724275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:07.724156 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:03:08.255339 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.255304 2570 generic.go:356] "Generic (PLEG): container finished" podID="8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" containerID="d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d" exitCode=0 Jul 21 21:03:08.255550 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.255349 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78994ff884-mvkv6" Jul 21 21:03:08.255550 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.255386 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78994ff884-mvkv6" event={"ID":"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3","Type":"ContainerDied","Data":"d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d"} Jul 21 21:03:08.255550 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.255436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78994ff884-mvkv6" event={"ID":"8e5bc631-a250-4f94-b3b3-218bb0e8f2b3","Type":"ContainerDied","Data":"3525d4dae88b6249baee546de4a6401663f1ac7d9110a3f87f414e96dcf552da"} Jul 21 21:03:08.255550 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.255461 2570 scope.go:117] "RemoveContainer" containerID="d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d" Jul 21 21:03:08.264009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.263993 2570 scope.go:117] "RemoveContainer" containerID="d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d" Jul 21 21:03:08.264248 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:03:08.264225 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d\": container with ID starting with d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d not found: ID does not exist" containerID="d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d" Jul 21 21:03:08.264337 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.264253 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d"} err="failed to get container status \"d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d\": rpc error: code = NotFound desc = could not find container \"d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d\": container with ID starting with d703d0586d13a0e6d3fff0528a9dc9c1d464d2b6e356fb44744471d46708e39d not found: ID does not exist" Jul 21 21:03:08.277030 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.277006 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78994ff884-mvkv6"] Jul 21 21:03:08.281671 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:08.281652 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78994ff884-mvkv6"] Jul 21 21:03:09.555165 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:09.555133 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" path="/var/lib/kubelet/pods/8e5bc631-a250-4f94-b3b3-218bb0e8f2b3/volumes" Jul 21 21:03:11.667927 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:11.667892 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:03:11.671633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:11.671608 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55f5b4445c-kp7vq" Jul 21 21:03:46.205913 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:46.205883 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:46.405064 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:46.404999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:52.315986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:52.315944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:55.564438 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.564405 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-796587bb68-77mbl"] Jul 21 21:03:55.564839 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.564695 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" containerName="console" Jul 21 21:03:55.564839 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.564705 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" containerName="console" Jul 21 21:03:55.564839 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.564764 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5bc631-a250-4f94-b3b3-218bb0e8f2b3" containerName="console" Jul 21 21:03:55.569940 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.569924 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.584012 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.583987 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796587bb68-77mbl"] Jul 21 21:03:55.635690 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.635664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:55.679719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679693 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-oauth-serving-cert\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.679860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679732 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-serving-cert\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.679860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhc6x\" (UniqueName: \"kubernetes.io/projected/9401624f-eb92-4d76-abde-454fada70605-kube-api-access-mhc6x\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.679938 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-trusted-ca-bundle\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.679938 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679895 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-service-ca\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.679938 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-console-config\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.680029 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.679942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-oauth-config\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.694569 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.694536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:03:55.780526 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780498 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-service-ca\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.780633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-console-config\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.780633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-oauth-config\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.780633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-oauth-serving-cert\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.780633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-serving-cert\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.780819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc6x\" (UniqueName: \"kubernetes.io/projected/9401624f-eb92-4d76-abde-454fada70605-kube-api-access-mhc6x\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.780819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.780700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-trusted-ca-bundle\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.781245 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.781220 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-service-ca\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.781402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.781356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-console-config\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.781459 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.781389 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-oauth-serving-cert\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.781817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.781795 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-trusted-ca-bundle\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.783018 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.782996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-oauth-config\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.783161 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.783142 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-serving-cert\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.788455 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.788433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhc6x\" (UniqueName: \"kubernetes.io/projected/9401624f-eb92-4d76-abde-454fada70605-kube-api-access-mhc6x\") pod \"console-796587bb68-77mbl\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:55.879169 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:55.879093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:03:56.008014 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:56.007990 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796587bb68-77mbl"] Jul 21 21:03:56.010092 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:03:56.010058 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9401624f_eb92_4d76_abde_454fada70605.slice/crio-b363ad6113890c2ddb7d5f36cab9a1b30aa142bee613b6a653803c132cfd10a9 WatchSource:0}: Error finding container b363ad6113890c2ddb7d5f36cab9a1b30aa142bee613b6a653803c132cfd10a9: Status 404 returned error can't find the container with id b363ad6113890c2ddb7d5f36cab9a1b30aa142bee613b6a653803c132cfd10a9 Jul 21 21:03:56.430770 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:56.430732 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796587bb68-77mbl" event={"ID":"9401624f-eb92-4d76-abde-454fada70605","Type":"ContainerStarted","Data":"812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685"} Jul 21 21:03:56.430946 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:56.430788 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796587bb68-77mbl" event={"ID":"9401624f-eb92-4d76-abde-454fada70605","Type":"ContainerStarted","Data":"b363ad6113890c2ddb7d5f36cab9a1b30aa142bee613b6a653803c132cfd10a9"} Jul 21 21:03:56.451239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:03:56.451188 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-796587bb68-77mbl" podStartSLOduration=1.451172642 podStartE2EDuration="1.451172642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:03:56.449742213 +0000 UTC m=+213.394363282" watchObservedRunningTime="2026-07-21 21:03:56.451172642 +0000 UTC m=+213.395793707" Jul 21 21:04:01.434574 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:01.434524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:01.500520 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:01.500491 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:05.571244 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.571214 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796587bb68-77mbl"] Jul 21 21:04:05.608262 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.608232 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85fdf46566-6v6w9"] Jul 21 21:04:05.611345 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.611326 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.622157 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.622133 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85fdf46566-6v6w9"] Jul 21 21:04:05.660399 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660354 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-oauth-serving-cert\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.660500 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-console-config\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.660500 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660431 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-oauth-config\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.660500 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-service-ca\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.660500 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-trusted-ca-bundle\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.660629 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-serving-cert\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.660629 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.660566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/3a3036da-ca81-406c-bfb8-eb5f15030631-kube-api-access-fwj9l\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-console-config\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-oauth-config\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-service-ca\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-trusted-ca-bundle\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-serving-cert\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/3a3036da-ca81-406c-bfb8-eb5f15030631-kube-api-access-fwj9l\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.761974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.761792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-oauth-serving-cert\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.762411 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.762387 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-console-config\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.762679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.762515 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-oauth-serving-cert\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.762679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.762519 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-service-ca\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.762679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.762592 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-trusted-ca-bundle\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.764343 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.764313 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-serving-cert\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.764462 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.764441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-oauth-config\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.771852 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.771832 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/3a3036da-ca81-406c-bfb8-eb5f15030631-kube-api-access-fwj9l\") pod \"console-85fdf46566-6v6w9\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:05.879738 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.879664 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:04:05.919965 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:05.919930 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:06.045248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:06.045227 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85fdf46566-6v6w9"] Jul 21 21:04:06.047820 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:04:06.047797 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3036da_ca81_406c_bfb8_eb5f15030631.slice/crio-6cc62f7c6a931095ae6ffce58fd744220d92cdf90723b085446f984c0c8c5831 WatchSource:0}: Error finding container 6cc62f7c6a931095ae6ffce58fd744220d92cdf90723b085446f984c0c8c5831: Status 404 returned error can't find the container with id 6cc62f7c6a931095ae6ffce58fd744220d92cdf90723b085446f984c0c8c5831 Jul 21 21:04:06.470540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:06.470501 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fdf46566-6v6w9" event={"ID":"3a3036da-ca81-406c-bfb8-eb5f15030631","Type":"ContainerStarted","Data":"ed20bea09f24fd1d04aeda7b32bcc4358a0a8aae0786024a9f38d3ed9f0249ad"} Jul 21 21:04:06.470693 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:06.470553 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fdf46566-6v6w9" event={"ID":"3a3036da-ca81-406c-bfb8-eb5f15030631","Type":"ContainerStarted","Data":"6cc62f7c6a931095ae6ffce58fd744220d92cdf90723b085446f984c0c8c5831"} Jul 21 21:04:06.490058 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:06.490009 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85fdf46566-6v6w9" podStartSLOduration=1.489994076 podStartE2EDuration="1.489994076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:04:06.489007191 +0000 UTC m=+223.433628253" watchObservedRunningTime="2026-07-21 21:04:06.489994076 +0000 UTC m=+223.434615141" Jul 21 21:04:13.443208 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:13.443169 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:13.506995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:13.506959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:15.920353 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:15.920319 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:15.920832 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:15.920399 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:15.925216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:15.925199 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:16.510505 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:16.510476 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:04:16.563748 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:16.563714 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546475b69b-c5w5l"] Jul 21 21:04:16.672018 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:16.671985 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:16.773041 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:16.772966 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:16.873327 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:16.873297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:19.091841 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:19.091806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:19.174311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:19.174276 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:19.570180 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:19.570147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:19.671738 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:19.671708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:21.357809 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:21.357771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:21.457897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:21.457865 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:04:30.590945 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.590910 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-796587bb68-77mbl" podUID="9401624f-eb92-4d76-abde-454fada70605" containerName="console" containerID="cri-o://812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685" gracePeriod=15 Jul 21 21:04:30.826203 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.826177 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:04:30.955239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955209 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-console-config\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955260 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-service-ca\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955282 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-oauth-serving-cert\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955300 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-oauth-config\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955331 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-serving-cert\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955380 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhc6x\" (UniqueName: \"kubernetes.io/projected/9401624f-eb92-4d76-abde-454fada70605-kube-api-access-mhc6x\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955420 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-trusted-ca-bundle\") pod \"9401624f-eb92-4d76-abde-454fada70605\" (UID: \"9401624f-eb92-4d76-abde-454fada70605\") " Jul 21 21:04:30.955735 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955566 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-console-config" (OuterVolumeSpecName: "console-config") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:30.955828 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955753 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:30.955948 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955832 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-service-ca" (OuterVolumeSpecName: "service-ca") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:30.956157 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.955957 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:30.957655 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.957630 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:04:30.957754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.957693 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:04:30.957864 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:30.957838 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9401624f-eb92-4d76-abde-454fada70605-kube-api-access-mhc6x" (OuterVolumeSpecName: "kube-api-access-mhc6x") pod "9401624f-eb92-4d76-abde-454fada70605" (UID: "9401624f-eb92-4d76-abde-454fada70605"). InnerVolumeSpecName "kube-api-access-mhc6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:04:31.057002 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.056969 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.057002 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.056996 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-oauth-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.057002 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.057006 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-oauth-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.057220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.057014 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9401624f-eb92-4d76-abde-454fada70605-console-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.057220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.057022 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhc6x\" (UniqueName: \"kubernetes.io/projected/9401624f-eb92-4d76-abde-454fada70605-kube-api-access-mhc6x\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.057220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.057030 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-trusted-ca-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.057220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.057038 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9401624f-eb92-4d76-abde-454fada70605-console-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:31.560476 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.560441 2570 generic.go:356] "Generic (PLEG): container finished" podID="9401624f-eb92-4d76-abde-454fada70605" containerID="812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685" exitCode=0 Jul 21 21:04:31.560647 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.560499 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796587bb68-77mbl" Jul 21 21:04:31.560647 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.560495 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796587bb68-77mbl" event={"ID":"9401624f-eb92-4d76-abde-454fada70605","Type":"ContainerDied","Data":"812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685"} Jul 21 21:04:31.560647 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.560626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796587bb68-77mbl" event={"ID":"9401624f-eb92-4d76-abde-454fada70605","Type":"ContainerDied","Data":"b363ad6113890c2ddb7d5f36cab9a1b30aa142bee613b6a653803c132cfd10a9"} Jul 21 21:04:31.560763 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.560657 2570 scope.go:117] "RemoveContainer" containerID="812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685" Jul 21 21:04:31.570222 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.570204 2570 scope.go:117] "RemoveContainer" containerID="812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685" Jul 21 21:04:31.570514 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:04:31.570494 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685\": container with ID starting with 812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685 not found: ID does not exist" containerID="812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685" Jul 21 21:04:31.570583 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.570527 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685"} err="failed to get container status \"812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685\": rpc error: code = NotFound desc = could not find container \"812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685\": container with ID starting with 812daba779f43bc77fa4ff14fc562de91a96d0d5342d43917bca4dcc0a649685 not found: ID does not exist" Jul 21 21:04:31.581988 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.581958 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796587bb68-77mbl"] Jul 21 21:04:31.583602 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:31.583571 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-796587bb68-77mbl"] Jul 21 21:04:33.555876 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:33.555843 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9401624f-eb92-4d76-abde-454fada70605" path="/var/lib/kubelet/pods/9401624f-eb92-4d76-abde-454fada70605/volumes" Jul 21 21:04:41.584762 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.584726 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-546475b69b-c5w5l" podUID="04b0a036-96d4-4a8d-9e86-89e48db73617" containerName="console" containerID="cri-o://2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6" gracePeriod=15 Jul 21 21:04:41.823592 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.823567 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:04:41.940788 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.940756 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlv6g\" (UniqueName: \"kubernetes.io/projected/04b0a036-96d4-4a8d-9e86-89e48db73617-kube-api-access-xlv6g\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.940934 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.940819 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-service-ca\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.940934 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.940858 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-oauth-config\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.941002 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.940971 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-trusted-ca-bundle\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.941033 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941016 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-serving-cert\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.941085 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941040 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-oauth-serving-cert\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.941085 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941068 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-console-config\") pod \"04b0a036-96d4-4a8d-9e86-89e48db73617\" (UID: \"04b0a036-96d4-4a8d-9e86-89e48db73617\") " Jul 21 21:04:41.941261 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941228 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-service-ca" (OuterVolumeSpecName: "service-ca") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:41.941410 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941351 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:41.941511 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941486 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:41.941679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.941646 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-console-config" (OuterVolumeSpecName: "console-config") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:04:41.943119 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.943091 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:04:41.943119 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.943104 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b0a036-96d4-4a8d-9e86-89e48db73617-kube-api-access-xlv6g" (OuterVolumeSpecName: "kube-api-access-xlv6g") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "kube-api-access-xlv6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:04:41.943119 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:41.943110 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04b0a036-96d4-4a8d-9e86-89e48db73617" (UID: "04b0a036-96d4-4a8d-9e86-89e48db73617"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:04:42.041898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041869 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-oauth-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.041898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041892 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-trusted-ca-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.041898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041901 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b0a036-96d4-4a8d-9e86-89e48db73617-console-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.042061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041909 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-oauth-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.042061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041919 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-console-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.042061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041927 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlv6g\" (UniqueName: \"kubernetes.io/projected/04b0a036-96d4-4a8d-9e86-89e48db73617-kube-api-access-xlv6g\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.042061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.041935 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b0a036-96d4-4a8d-9e86-89e48db73617-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:04:42.600075 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.600043 2570 generic.go:356] "Generic (PLEG): container finished" podID="04b0a036-96d4-4a8d-9e86-89e48db73617" containerID="2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6" exitCode=0 Jul 21 21:04:42.600472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.600086 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546475b69b-c5w5l" Jul 21 21:04:42.600472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.600130 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546475b69b-c5w5l" event={"ID":"04b0a036-96d4-4a8d-9e86-89e48db73617","Type":"ContainerDied","Data":"2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6"} Jul 21 21:04:42.600472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.600183 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546475b69b-c5w5l" event={"ID":"04b0a036-96d4-4a8d-9e86-89e48db73617","Type":"ContainerDied","Data":"3cdbe6a4951ed22583edb49f00ef71481cbae2d3e03e9351d8bc8685e979c44b"} Jul 21 21:04:42.600472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.600209 2570 scope.go:117] "RemoveContainer" containerID="2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6" Jul 21 21:04:42.608911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.608895 2570 scope.go:117] "RemoveContainer" containerID="2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6" Jul 21 21:04:42.609150 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:04:42.609130 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6\": container with ID starting with 2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6 not found: ID does not exist" containerID="2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6" Jul 21 21:04:42.609190 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.609159 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6"} err="failed to get container status \"2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6\": rpc error: code = NotFound desc = could not find container \"2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6\": container with ID starting with 2608efd14fc552820c9e4885c72990c676ec84f85f6f19c86846b2b9eb03afe6 not found: ID does not exist" Jul 21 21:04:42.625214 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.622411 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546475b69b-c5w5l"] Jul 21 21:04:42.626338 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:42.626313 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546475b69b-c5w5l"] Jul 21 21:04:43.555349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:04:43.555314 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b0a036-96d4-4a8d-9e86-89e48db73617" path="/var/lib/kubelet/pods/04b0a036-96d4-4a8d-9e86-89e48db73617/volumes" Jul 21 21:05:01.745999 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:01.745892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:09.527353 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:09.527321 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:09.614799 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:09.614765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:10.260599 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260568 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2"] Jul 21 21:05:10.260890 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260876 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b0a036-96d4-4a8d-9e86-89e48db73617" containerName="console" Jul 21 21:05:10.260890 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260891 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b0a036-96d4-4a8d-9e86-89e48db73617" containerName="console" Jul 21 21:05:10.261031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260901 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9401624f-eb92-4d76-abde-454fada70605" containerName="console" Jul 21 21:05:10.261031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260906 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9401624f-eb92-4d76-abde-454fada70605" containerName="console" Jul 21 21:05:10.261031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260963 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b0a036-96d4-4a8d-9e86-89e48db73617" containerName="console" Jul 21 21:05:10.261031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.260970 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9401624f-eb92-4d76-abde-454fada70605" containerName="console" Jul 21 21:05:10.264287 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.264267 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.267570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.267549 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 21:05:10.267666 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.267608 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 21:05:10.269188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.269171 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8ws\"" Jul 21 21:05:10.273819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.273798 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2"] Jul 21 21:05:10.343645 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.343608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ktr\" (UniqueName: \"kubernetes.io/projected/4fc24cf8-60bd-4cf2-9252-256deb2e5965-kube-api-access-x2ktr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.343804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.343669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.343804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.343708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.444508 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.444469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.444669 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.444516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.444669 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.444564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ktr\" (UniqueName: \"kubernetes.io/projected/4fc24cf8-60bd-4cf2-9252-256deb2e5965-kube-api-access-x2ktr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.444874 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.444853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.444918 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.444881 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.453896 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.453867 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ktr\" (UniqueName: \"kubernetes.io/projected/4fc24cf8-60bd-4cf2-9252-256deb2e5965-kube-api-access-x2ktr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.573260 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.573177 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:10.702711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:10.702678 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2"] Jul 21 21:05:10.706509 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:05:10.706467 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc24cf8_60bd_4cf2_9252_256deb2e5965.slice/crio-c33b880a4be62577bbb42bcb33770597b46454da17f792162afed0fd9576c594 WatchSource:0}: Error finding container c33b880a4be62577bbb42bcb33770597b46454da17f792162afed0fd9576c594: Status 404 returned error can't find the container with id c33b880a4be62577bbb42bcb33770597b46454da17f792162afed0fd9576c594 Jul 21 21:05:11.706470 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:11.706434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" event={"ID":"4fc24cf8-60bd-4cf2-9252-256deb2e5965","Type":"ContainerStarted","Data":"c33b880a4be62577bbb42bcb33770597b46454da17f792162afed0fd9576c594"} Jul 21 21:05:17.731006 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:17.730969 2570 generic.go:356] "Generic (PLEG): container finished" podID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerID="4bc3d091f8b13e0391ca407296784c14bd0f92675a7fe86f5e84eba6d3a825b2" exitCode=0 Jul 21 21:05:17.731404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:17.731050 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" event={"ID":"4fc24cf8-60bd-4cf2-9252-256deb2e5965","Type":"ContainerDied","Data":"4bc3d091f8b13e0391ca407296784c14bd0f92675a7fe86f5e84eba6d3a825b2"} Jul 21 21:05:20.743895 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:20.743861 2570 generic.go:356] "Generic (PLEG): container finished" podID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerID="6028b8a4f8a5c7811c6b5911f3ca8b4e51b1f91cdd46917348c5997856903b33" exitCode=0 Jul 21 21:05:20.744247 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:20.743935 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" event={"ID":"4fc24cf8-60bd-4cf2-9252-256deb2e5965","Type":"ContainerDied","Data":"6028b8a4f8a5c7811c6b5911f3ca8b4e51b1f91cdd46917348c5997856903b33"} Jul 21 21:05:22.148351 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:22.148313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:22.240430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:22.240398 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:23.463975 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:23.463949 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 21:05:28.778165 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:28.778124 2570 generic.go:356] "Generic (PLEG): container finished" podID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerID="f08841b5065ca17c5f6cb0899b2d1084222f43a6056f4080721e4c6021bb93f8" exitCode=0 Jul 21 21:05:28.778590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:28.778174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" event={"ID":"4fc24cf8-60bd-4cf2-9252-256deb2e5965","Type":"ContainerDied","Data":"f08841b5065ca17c5f6cb0899b2d1084222f43a6056f4080721e4c6021bb93f8"} Jul 21 21:05:29.934283 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:29.934254 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:30.009717 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.009687 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-bundle\") pod \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " Jul 21 21:05:30.009865 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.009790 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-util\") pod \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " Jul 21 21:05:30.009865 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.009829 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2ktr\" (UniqueName: \"kubernetes.io/projected/4fc24cf8-60bd-4cf2-9252-256deb2e5965-kube-api-access-x2ktr\") pod \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\" (UID: \"4fc24cf8-60bd-4cf2-9252-256deb2e5965\") " Jul 21 21:05:30.010443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.010415 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-bundle" (OuterVolumeSpecName: "bundle") pod "4fc24cf8-60bd-4cf2-9252-256deb2e5965" (UID: "4fc24cf8-60bd-4cf2-9252-256deb2e5965"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:05:30.012043 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.012017 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc24cf8-60bd-4cf2-9252-256deb2e5965-kube-api-access-x2ktr" (OuterVolumeSpecName: "kube-api-access-x2ktr") pod "4fc24cf8-60bd-4cf2-9252-256deb2e5965" (UID: "4fc24cf8-60bd-4cf2-9252-256deb2e5965"). InnerVolumeSpecName "kube-api-access-x2ktr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:05:30.014085 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.014063 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-util" (OuterVolumeSpecName: "util") pod "4fc24cf8-60bd-4cf2-9252-256deb2e5965" (UID: "4fc24cf8-60bd-4cf2-9252-256deb2e5965"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:05:30.111258 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.111187 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:05:30.111258 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.111214 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fc24cf8-60bd-4cf2-9252-256deb2e5965-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:05:30.111258 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.111224 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2ktr\" (UniqueName: \"kubernetes.io/projected/4fc24cf8-60bd-4cf2-9252-256deb2e5965-kube-api-access-x2ktr\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:05:30.787856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.787827 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" Jul 21 21:05:30.787856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.787842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754024pg2" event={"ID":"4fc24cf8-60bd-4cf2-9252-256deb2e5965","Type":"ContainerDied","Data":"c33b880a4be62577bbb42bcb33770597b46454da17f792162afed0fd9576c594"} Jul 21 21:05:30.788105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:30.787883 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33b880a4be62577bbb42bcb33770597b46454da17f792162afed0fd9576c594" Jul 21 21:05:36.625834 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:36.625803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:38.060542 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.060509 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:38.823259 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823228 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh"] Jul 21 21:05:38.823551 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823535 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="util" Jul 21 21:05:38.823597 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823552 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="util" Jul 21 21:05:38.823597 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823564 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="extract" Jul 21 21:05:38.823597 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823569 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="extract" Jul 21 21:05:38.823597 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823580 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="pull" Jul 21 21:05:38.823597 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823586 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="pull" Jul 21 21:05:38.823741 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.823629 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc24cf8-60bd-4cf2-9252-256deb2e5965" containerName="extract" Jul 21 21:05:38.826897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.826866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:38.830299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.830273 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:05:38.830447 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.830273 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 21:05:38.830447 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.830285 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4t6dh\"" Jul 21 21:05:38.842091 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.842064 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh"] Jul 21 21:05:38.976098 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.976060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c989c4-5d46-4297-8150-3fa1f1bc31ac-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x2dlh\" (UID: \"63c989c4-5d46-4297-8150-3fa1f1bc31ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:38.976265 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:38.976120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rbv\" (UniqueName: \"kubernetes.io/projected/63c989c4-5d46-4297-8150-3fa1f1bc31ac-kube-api-access-s4rbv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x2dlh\" (UID: \"63c989c4-5d46-4297-8150-3fa1f1bc31ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:39.077322 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.077244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c989c4-5d46-4297-8150-3fa1f1bc31ac-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x2dlh\" (UID: \"63c989c4-5d46-4297-8150-3fa1f1bc31ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:39.077322 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.077312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rbv\" (UniqueName: \"kubernetes.io/projected/63c989c4-5d46-4297-8150-3fa1f1bc31ac-kube-api-access-s4rbv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x2dlh\" (UID: \"63c989c4-5d46-4297-8150-3fa1f1bc31ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:39.077769 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.077638 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c989c4-5d46-4297-8150-3fa1f1bc31ac-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x2dlh\" (UID: \"63c989c4-5d46-4297-8150-3fa1f1bc31ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:39.086908 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.086882 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rbv\" (UniqueName: \"kubernetes.io/projected/63c989c4-5d46-4297-8150-3fa1f1bc31ac-kube-api-access-s4rbv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x2dlh\" (UID: \"63c989c4-5d46-4297-8150-3fa1f1bc31ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:39.136865 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.136830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" Jul 21 21:05:39.280433 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.280403 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh"] Jul 21 21:05:39.284288 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:05:39.284261 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c989c4_5d46_4297_8150_3fa1f1bc31ac.slice/crio-090179abdb3bbc289d83aaed110850ebcb79b40f22d080c847781d279b7b521f WatchSource:0}: Error finding container 090179abdb3bbc289d83aaed110850ebcb79b40f22d080c847781d279b7b521f: Status 404 returned error can't find the container with id 090179abdb3bbc289d83aaed110850ebcb79b40f22d080c847781d279b7b521f Jul 21 21:05:39.287580 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.287565 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:05:39.824970 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:39.824933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" event={"ID":"63c989c4-5d46-4297-8150-3fa1f1bc31ac","Type":"ContainerStarted","Data":"090179abdb3bbc289d83aaed110850ebcb79b40f22d080c847781d279b7b521f"} Jul 21 21:05:40.252580 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:40.252527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:40.352567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:40.352535 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:41.745644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:41.745614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:41.836305 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:41.836268 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" event={"ID":"63c989c4-5d46-4297-8150-3fa1f1bc31ac","Type":"ContainerStarted","Data":"b78e76d2ef6509f347fe1ff5c3aa17c282029b5d11f4fad99f2658193dcb569a"} Jul 21 21:05:41.873834 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:41.873804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:41.875637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:41.875584 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x2dlh" podStartSLOduration=2.081714439 podStartE2EDuration="3.875566748s" totalImagesPullingTime="1.793852309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:05:41.874324347 +0000 UTC m=+318.818945422" watchObservedRunningTime="2026-07-21 21:05:41.875566748 +0000 UTC m=+318.820187814" Jul 21 21:05:42.377267 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:42.377239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:42.480197 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:42.480150 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:43.065200 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.065169 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd"] Jul 21 21:05:43.067850 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.067833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.070578 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.070551 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8ws\"" Jul 21 21:05:43.070703 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.070678 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 21:05:43.071608 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.071588 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 21:05:43.077436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.077413 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd"] Jul 21 21:05:43.213589 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.213547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpg82\" (UniqueName: \"kubernetes.io/projected/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-kube-api-access-zpg82\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.213793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.213596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.213793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.213726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.314899 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.314859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpg82\" (UniqueName: \"kubernetes.io/projected/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-kube-api-access-zpg82\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.314899 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.314904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.315233 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.314947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.315302 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.315261 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.315302 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.315291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.323837 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.323810 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpg82\" (UniqueName: \"kubernetes.io/projected/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-kube-api-access-zpg82\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.377816 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.377788 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:43.514516 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.514488 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd"] Jul 21 21:05:43.516591 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:05:43.516561 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e31a5f8_3daf_4130_b718_8de9e4d15cfe.slice/crio-9d6dcf2155003ee4c89f758f2688ee9279f3ec805c9a4df38047773a645d3591 WatchSource:0}: Error finding container 9d6dcf2155003ee4c89f758f2688ee9279f3ec805c9a4df38047773a645d3591: Status 404 returned error can't find the container with id 9d6dcf2155003ee4c89f758f2688ee9279f3ec805c9a4df38047773a645d3591 Jul 21 21:05:43.846180 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.846091 2570 generic.go:356] "Generic (PLEG): container finished" podID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerID="ef826d3f24fd305435045dd00e231364cded968883c7b0877f4c82c80f9a510d" exitCode=0 Jul 21 21:05:43.846180 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.846168 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" event={"ID":"8e31a5f8-3daf-4130-b718-8de9e4d15cfe","Type":"ContainerDied","Data":"ef826d3f24fd305435045dd00e231364cded968883c7b0877f4c82c80f9a510d"} Jul 21 21:05:43.846377 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:43.846218 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" event={"ID":"8e31a5f8-3daf-4130-b718-8de9e4d15cfe","Type":"ContainerStarted","Data":"9d6dcf2155003ee4c89f758f2688ee9279f3ec805c9a4df38047773a645d3591"} Jul 21 21:05:44.141689 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.141610 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:44.379215 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.379181 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f9zxr"] Jul 21 21:05:44.381268 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.381252 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.386258 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.386227 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 21:05:44.386388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.386312 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jxxpn\"" Jul 21 21:05:44.390954 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.390938 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 21:05:44.413763 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.413740 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f9zxr"] Jul 21 21:05:44.523438 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.523407 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvck\" (UniqueName: \"kubernetes.io/projected/4722cd7c-bd43-4555-a2ab-ccd618ed45f5-kube-api-access-8wvck\") pod \"cert-manager-webhook-6486ffd997-f9zxr\" (UID: \"4722cd7c-bd43-4555-a2ab-ccd618ed45f5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.523596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.523454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4722cd7c-bd43-4555-a2ab-ccd618ed45f5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f9zxr\" (UID: \"4722cd7c-bd43-4555-a2ab-ccd618ed45f5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.602786 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.602752 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:44.624716 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.624687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvck\" (UniqueName: \"kubernetes.io/projected/4722cd7c-bd43-4555-a2ab-ccd618ed45f5-kube-api-access-8wvck\") pod \"cert-manager-webhook-6486ffd997-f9zxr\" (UID: \"4722cd7c-bd43-4555-a2ab-ccd618ed45f5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.624857 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.624740 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4722cd7c-bd43-4555-a2ab-ccd618ed45f5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f9zxr\" (UID: \"4722cd7c-bd43-4555-a2ab-ccd618ed45f5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.634024 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.633995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4722cd7c-bd43-4555-a2ab-ccd618ed45f5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f9zxr\" (UID: \"4722cd7c-bd43-4555-a2ab-ccd618ed45f5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.634117 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.634098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvck\" (UniqueName: \"kubernetes.io/projected/4722cd7c-bd43-4555-a2ab-ccd618ed45f5-kube-api-access-8wvck\") pod \"cert-manager-webhook-6486ffd997-f9zxr\" (UID: \"4722cd7c-bd43-4555-a2ab-ccd618ed45f5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.699229 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.699149 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:44.874476 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:44.874440 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f9zxr"] Jul 21 21:05:44.877744 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:05:44.877718 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4722cd7c_bd43_4555_a2ab_ccd618ed45f5.slice/crio-3f2dc82b58eabb48db455c715b2fe06206dae461e86e8a1bae96daba6be8071c WatchSource:0}: Error finding container 3f2dc82b58eabb48db455c715b2fe06206dae461e86e8a1bae96daba6be8071c: Status 404 returned error can't find the container with id 3f2dc82b58eabb48db455c715b2fe06206dae461e86e8a1bae96daba6be8071c Jul 21 21:05:45.859956 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:45.859913 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" event={"ID":"4722cd7c-bd43-4555-a2ab-ccd618ed45f5","Type":"ContainerStarted","Data":"3f2dc82b58eabb48db455c715b2fe06206dae461e86e8a1bae96daba6be8071c"} Jul 21 21:05:46.867141 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:46.867103 2570 generic.go:356] "Generic (PLEG): container finished" podID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerID="d506535c4f90911fbee8f1196093734c6292f2818a85d21cd20c1a26c36f794a" exitCode=0 Jul 21 21:05:46.867141 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:46.867143 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" event={"ID":"8e31a5f8-3daf-4130-b718-8de9e4d15cfe","Type":"ContainerDied","Data":"d506535c4f90911fbee8f1196093734c6292f2818a85d21cd20c1a26c36f794a"} Jul 21 21:05:47.247324 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:47.247269 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:47.345052 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:47.345013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:47.877575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:47.877534 2570 generic.go:356] "Generic (PLEG): container finished" podID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerID="bf16417e976aa2e2d852d970f4da826a4997ebb159581f70e550026ea670c420" exitCode=0 Jul 21 21:05:47.878018 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:47.877603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" event={"ID":"8e31a5f8-3daf-4130-b718-8de9e4d15cfe","Type":"ContainerDied","Data":"bf16417e976aa2e2d852d970f4da826a4997ebb159581f70e550026ea670c420"} Jul 21 21:05:48.883570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:48.883523 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" event={"ID":"4722cd7c-bd43-4555-a2ab-ccd618ed45f5","Type":"ContainerStarted","Data":"901583cb85e2051fab398eea45d052ee618e5b8dbf85cbbc306cb823f8c2d272"} Jul 21 21:05:48.884109 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:48.883773 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:48.904767 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:48.904706 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" podStartSLOduration=1.3960705660000001 podStartE2EDuration="4.904689688s" totalImagesPullingTime="3.508619122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:05:48.904332295 +0000 UTC m=+325.848953385" watchObservedRunningTime="2026-07-21 21:05:48.904689688 +0000 UTC m=+325.849310753" Jul 21 21:05:49.015826 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.015800 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:49.162112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.162026 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-bundle\") pod \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " Jul 21 21:05:49.162112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.162066 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-util\") pod \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " Jul 21 21:05:49.162112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.162105 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpg82\" (UniqueName: \"kubernetes.io/projected/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-kube-api-access-zpg82\") pod \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\" (UID: \"8e31a5f8-3daf-4130-b718-8de9e4d15cfe\") " Jul 21 21:05:49.162469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.162447 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-bundle" (OuterVolumeSpecName: "bundle") pod "8e31a5f8-3daf-4130-b718-8de9e4d15cfe" (UID: "8e31a5f8-3daf-4130-b718-8de9e4d15cfe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:05:49.164348 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.164326 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-kube-api-access-zpg82" (OuterVolumeSpecName: "kube-api-access-zpg82") pod "8e31a5f8-3daf-4130-b718-8de9e4d15cfe" (UID: "8e31a5f8-3daf-4130-b718-8de9e4d15cfe"). InnerVolumeSpecName "kube-api-access-zpg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:05:49.168882 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.168852 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-util" (OuterVolumeSpecName: "util") pod "8e31a5f8-3daf-4130-b718-8de9e4d15cfe" (UID: "8e31a5f8-3daf-4130-b718-8de9e4d15cfe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:05:49.262817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.262787 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpg82\" (UniqueName: \"kubernetes.io/projected/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-kube-api-access-zpg82\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:05:49.262817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.262810 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:05:49.262817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.262818 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e31a5f8-3daf-4130-b718-8de9e4d15cfe-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:05:49.369985 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.369948 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:49.439280 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.439253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:49.889627 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.889533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" event={"ID":"8e31a5f8-3daf-4130-b718-8de9e4d15cfe","Type":"ContainerDied","Data":"9d6dcf2155003ee4c89f758f2688ee9279f3ec805c9a4df38047773a645d3591"} Jul 21 21:05:49.889627 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.889580 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6dcf2155003ee4c89f758f2688ee9279f3ec805c9a4df38047773a645d3591" Jul 21 21:05:49.890053 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:49.889682 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjdcxd" Jul 21 21:05:52.357971 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:52.357936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:53.647343 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.647304 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:53.738998 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.738961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:53.870605 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870572 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4jscg"] Jul 21 21:05:53.870857 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870845 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="extract" Jul 21 21:05:53.870905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870858 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="extract" Jul 21 21:05:53.870905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870870 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="util" Jul 21 21:05:53.870905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870876 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="util" Jul 21 21:05:53.870905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870887 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="pull" Jul 21 21:05:53.870905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870892 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="pull" Jul 21 21:05:53.871052 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.870940 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e31a5f8-3daf-4130-b718-8de9e4d15cfe" containerName="extract" Jul 21 21:05:53.872735 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.872720 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:53.875259 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.875242 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c524t\"" Jul 21 21:05:53.883924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.883900 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4jscg"] Jul 21 21:05:53.904025 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.903968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdd9p\" (UniqueName: \"kubernetes.io/projected/fde0ebc6-5b39-4972-8ce1-bba7a0a282b1-kube-api-access-cdd9p\") pod \"cert-manager-55b4d954d5-4jscg\" (UID: \"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1\") " pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:53.904025 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:53.904010 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fde0ebc6-5b39-4972-8ce1-bba7a0a282b1-bound-sa-token\") pod \"cert-manager-55b4d954d5-4jscg\" (UID: \"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1\") " pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:54.005212 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.005180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fde0ebc6-5b39-4972-8ce1-bba7a0a282b1-bound-sa-token\") pod \"cert-manager-55b4d954d5-4jscg\" (UID: \"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1\") " pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:54.005381 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.005248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdd9p\" (UniqueName: \"kubernetes.io/projected/fde0ebc6-5b39-4972-8ce1-bba7a0a282b1-kube-api-access-cdd9p\") pod \"cert-manager-55b4d954d5-4jscg\" (UID: \"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1\") " pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:54.013105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.013079 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fde0ebc6-5b39-4972-8ce1-bba7a0a282b1-bound-sa-token\") pod \"cert-manager-55b4d954d5-4jscg\" (UID: \"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1\") " pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:54.013402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.013352 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdd9p\" (UniqueName: \"kubernetes.io/projected/fde0ebc6-5b39-4972-8ce1-bba7a0a282b1-kube-api-access-cdd9p\") pod \"cert-manager-55b4d954d5-4jscg\" (UID: \"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1\") " pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:54.181281 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.181247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4jscg" Jul 21 21:05:54.330356 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.330332 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4jscg"] Jul 21 21:05:54.331687 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:05:54.331657 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde0ebc6_5b39_4972_8ce1_bba7a0a282b1.slice/crio-da2cc34c49794fe37f8b887327ef89dc0de54ef130f922a4ca8a793390786975 WatchSource:0}: Error finding container da2cc34c49794fe37f8b887327ef89dc0de54ef130f922a4ca8a793390786975: Status 404 returned error can't find the container with id da2cc34c49794fe37f8b887327ef89dc0de54ef130f922a4ca8a793390786975 Jul 21 21:05:54.892459 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.892427 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f9zxr" Jul 21 21:05:54.916270 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.916235 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4jscg" event={"ID":"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1","Type":"ContainerStarted","Data":"2cd081a6e4c5911dd371778a096c6b93c5408687cc2262fc9e897c24248f9461"} Jul 21 21:05:54.916270 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.916273 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4jscg" event={"ID":"fde0ebc6-5b39-4972-8ce1-bba7a0a282b1","Type":"ContainerStarted","Data":"da2cc34c49794fe37f8b887327ef89dc0de54ef130f922a4ca8a793390786975"} Jul 21 21:05:54.933481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:54.933436 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4jscg" podStartSLOduration=1.933418713 podStartE2EDuration="1.933418713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:05:54.931959256 +0000 UTC m=+331.876580343" watchObservedRunningTime="2026-07-21 21:05:54.933418713 +0000 UTC m=+331.878039780" Jul 21 21:05:59.779666 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:59.779628 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:05:59.882113 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:05:59.882082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:01.731566 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:01.731533 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:02.772380 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.772330 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd"] Jul 21 21:06:02.814263 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.814224 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd"] Jul 21 21:06:02.814430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.814320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.817442 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.817422 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 21:06:02.817541 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.817440 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8ws\"" Jul 21 21:06:02.817541 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.817422 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 21:06:02.874467 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.874435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rz7\" (UniqueName: \"kubernetes.io/projected/8209c401-dd86-4a5a-8ec3-4b5e104583e4-kube-api-access-z9rz7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.874636 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.874502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.874636 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.874530 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.974883 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.974842 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rz7\" (UniqueName: \"kubernetes.io/projected/8209c401-dd86-4a5a-8ec3-4b5e104583e4-kube-api-access-z9rz7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.975074 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.974906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.975074 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.974939 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.975298 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.975279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.975385 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.975315 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:02.984147 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:02.984125 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rz7\" (UniqueName: \"kubernetes.io/projected/8209c401-dd86-4a5a-8ec3-4b5e104583e4-kube-api-access-z9rz7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:03.123585 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:03.123497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:03.274354 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:03.274301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd"] Jul 21 21:06:03.276922 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:03.276871 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8209c401_dd86_4a5a_8ec3_4b5e104583e4.slice/crio-8fa4f1f786b2096cf0ee43815748e88facc17f4390c8309d51004db176ad6bcc WatchSource:0}: Error finding container 8fa4f1f786b2096cf0ee43815748e88facc17f4390c8309d51004db176ad6bcc: Status 404 returned error can't find the container with id 8fa4f1f786b2096cf0ee43815748e88facc17f4390c8309d51004db176ad6bcc Jul 21 21:06:03.952341 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:03.952306 2570 generic.go:356] "Generic (PLEG): container finished" podID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerID="28930232b33b2a9029ba93f1ecab4efae9de5b57c8805d6537da24a53a37f7f1" exitCode=0 Jul 21 21:06:03.952706 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:03.952398 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" event={"ID":"8209c401-dd86-4a5a-8ec3-4b5e104583e4","Type":"ContainerDied","Data":"28930232b33b2a9029ba93f1ecab4efae9de5b57c8805d6537da24a53a37f7f1"} Jul 21 21:06:03.952706 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:03.952443 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" event={"ID":"8209c401-dd86-4a5a-8ec3-4b5e104583e4","Type":"ContainerStarted","Data":"8fa4f1f786b2096cf0ee43815748e88facc17f4390c8309d51004db176ad6bcc"} Jul 21 21:06:04.958105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:04.958070 2570 generic.go:356] "Generic (PLEG): container finished" podID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerID="3952fcf8b643064907ea10ec326e50654a5083618b486aef33b3178b5d16176e" exitCode=0 Jul 21 21:06:04.958570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:04.958169 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" event={"ID":"8209c401-dd86-4a5a-8ec3-4b5e104583e4","Type":"ContainerDied","Data":"3952fcf8b643064907ea10ec326e50654a5083618b486aef33b3178b5d16176e"} Jul 21 21:06:05.964215 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:05.964177 2570 generic.go:356] "Generic (PLEG): container finished" podID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerID="d9b3b44e72e61b677688c8942e77e6cb5eb7122d2d7e3eed7b809f29fccec692" exitCode=0 Jul 21 21:06:05.964625 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:05.964261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" event={"ID":"8209c401-dd86-4a5a-8ec3-4b5e104583e4","Type":"ContainerDied","Data":"d9b3b44e72e61b677688c8942e77e6cb5eb7122d2d7e3eed7b809f29fccec692"} Jul 21 21:06:07.097494 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.097465 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:07.209916 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.209887 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-bundle\") pod \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " Jul 21 21:06:07.210070 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.209930 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-util\") pod \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " Jul 21 21:06:07.210070 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.209978 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9rz7\" (UniqueName: \"kubernetes.io/projected/8209c401-dd86-4a5a-8ec3-4b5e104583e4-kube-api-access-z9rz7\") pod \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\" (UID: \"8209c401-dd86-4a5a-8ec3-4b5e104583e4\") " Jul 21 21:06:07.210676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.210650 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-bundle" (OuterVolumeSpecName: "bundle") pod "8209c401-dd86-4a5a-8ec3-4b5e104583e4" (UID: "8209c401-dd86-4a5a-8ec3-4b5e104583e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:06:07.212101 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.212082 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8209c401-dd86-4a5a-8ec3-4b5e104583e4-kube-api-access-z9rz7" (OuterVolumeSpecName: "kube-api-access-z9rz7") pod "8209c401-dd86-4a5a-8ec3-4b5e104583e4" (UID: "8209c401-dd86-4a5a-8ec3-4b5e104583e4"). InnerVolumeSpecName "kube-api-access-z9rz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:06:07.215441 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.215403 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-util" (OuterVolumeSpecName: "util") pod "8209c401-dd86-4a5a-8ec3-4b5e104583e4" (UID: "8209c401-dd86-4a5a-8ec3-4b5e104583e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:06:07.311389 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.311286 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:06:07.311389 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.311312 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8209c401-dd86-4a5a-8ec3-4b5e104583e4-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:06:07.311389 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.311321 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rz7\" (UniqueName: \"kubernetes.io/projected/8209c401-dd86-4a5a-8ec3-4b5e104583e4-kube-api-access-z9rz7\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:06:07.974676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.974649 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" Jul 21 21:06:07.974833 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.974675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af75ktd" event={"ID":"8209c401-dd86-4a5a-8ec3-4b5e104583e4","Type":"ContainerDied","Data":"8fa4f1f786b2096cf0ee43815748e88facc17f4390c8309d51004db176ad6bcc"} Jul 21 21:06:07.974833 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:07.974718 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa4f1f786b2096cf0ee43815748e88facc17f4390c8309d51004db176ad6bcc" Jul 21 21:06:13.674824 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:13.674792 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:13.756676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:13.756645 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:14.438194 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.438162 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:14.775711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.775615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:14.880439 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.880398 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:14.904760 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.904734 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd"] Jul 21 21:06:14.905083 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905069 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="util" Jul 21 21:06:14.905129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905085 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="util" Jul 21 21:06:14.905129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905097 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="pull" Jul 21 21:06:14.905129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905102 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="pull" Jul 21 21:06:14.905129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905109 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="extract" Jul 21 21:06:14.905129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905114 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="extract" Jul 21 21:06:14.905284 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.905163 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8209c401-dd86-4a5a-8ec3-4b5e104583e4" containerName="extract" Jul 21 21:06:14.907014 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.906997 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:14.910041 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.910009 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 21:06:14.910041 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.910014 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 21:06:14.910212 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.910155 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 21:06:14.910598 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.910580 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ddkqb\"" Jul 21 21:06:14.910961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.910946 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 21:06:14.929494 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.929472 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd"] Jul 21 21:06:14.969756 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.969727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnd2\" (UniqueName: \"kubernetes.io/projected/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-kube-api-access-rqnd2\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:14.969864 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.969762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:14.969864 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.969782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-tmp\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:14.969864 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.969811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-webhook-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:14.969864 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:14.969858 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-manifests\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.035593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.035528 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:15.071217 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.071197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.071308 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.071225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-tmp\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.071308 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.071251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-webhook-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.071418 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.071396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-manifests\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.071496 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.071481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnd2\" (UniqueName: \"kubernetes.io/projected/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-kube-api-access-rqnd2\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.071801 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.071780 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-manifests\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.073656 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.073635 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-tmp\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.073757 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.073733 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.073817 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.073768 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-webhook-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.079741 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.079718 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnd2\" (UniqueName: \"kubernetes.io/projected/5258cfd5-86d0-4592-b8bc-2ea861d1a35d-kube-api-access-rqnd2\") pod \"opendatahub-operator-controller-manager-694dcbb678-rrjnd\" (UID: \"5258cfd5-86d0-4592-b8bc-2ea861d1a35d\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.136283 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.136224 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:15.216448 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.216423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:15.253674 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.253644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:15.351507 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.351485 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd"] Jul 21 21:06:15.353698 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:15.353667 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5258cfd5_86d0_4592_b8bc_2ea861d1a35d.slice/crio-647153451d13c51f2a1158b83e8beffee7f337f7d66fa550c9dd317d5e5997c5 WatchSource:0}: Error finding container 647153451d13c51f2a1158b83e8beffee7f337f7d66fa550c9dd317d5e5997c5: Status 404 returned error can't find the container with id 647153451d13c51f2a1158b83e8beffee7f337f7d66fa550c9dd317d5e5997c5 Jul 21 21:06:15.357164 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.357134 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:15.425409 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.425381 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:15.529083 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.529056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:15.629660 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:15.629583 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:16.013991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:16.013946 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" event={"ID":"5258cfd5-86d0-4592-b8bc-2ea861d1a35d","Type":"ContainerStarted","Data":"647153451d13c51f2a1158b83e8beffee7f337f7d66fa550c9dd317d5e5997c5"} Jul 21 21:06:19.029506 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:19.029471 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" event={"ID":"5258cfd5-86d0-4592-b8bc-2ea861d1a35d","Type":"ContainerStarted","Data":"3451809bb371871ea718e289277998f76ec690cdfa8c33a338013676fbbdc147"} Jul 21 21:06:20.022930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:20.022898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:20.034932 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:20.034905 2570 generic.go:356] "Generic (PLEG): container finished" podID="5258cfd5-86d0-4592-b8bc-2ea861d1a35d" containerID="3451809bb371871ea718e289277998f76ec690cdfa8c33a338013676fbbdc147" exitCode=0 Jul 21 21:06:20.035297 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:20.034999 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" event={"ID":"5258cfd5-86d0-4592-b8bc-2ea861d1a35d","Type":"ContainerDied","Data":"3451809bb371871ea718e289277998f76ec690cdfa8c33a338013676fbbdc147"} Jul 21 21:06:20.124262 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:20.124230 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:21.041327 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:21.041291 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" event={"ID":"5258cfd5-86d0-4592-b8bc-2ea861d1a35d","Type":"ContainerStarted","Data":"1168c0e65ff35e360fc715f28c077813d0aef7a2ae10746c99041023093bfb61"} Jul 21 21:06:21.046489 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:21.041781 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:21.071344 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:21.071286 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" podStartSLOduration=2.5227205870000002 podStartE2EDuration="7.071273406s" totalImagesPullingTime="3.520912217s" totalInitContainerRuntime="1.027640602s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:21.069427858 +0000 UTC m=+358.014048923" watchObservedRunningTime="2026-07-21 21:06:21.071273406 +0000 UTC m=+358.015894534" Jul 21 21:06:32.047856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:32.047826 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-rrjnd" Jul 21 21:06:34.023947 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.023914 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:34.204229 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.204189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:34.766660 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.766623 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fkrdd"] Jul 21 21:06:34.772752 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.772729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:34.775893 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.775872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j66dh\"" Jul 21 21:06:34.776123 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.776103 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 21:06:34.782420 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.782392 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fkrdd"] Jul 21 21:06:34.812633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.812603 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:34.903881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.903846 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:34.941720 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.941681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516b1e47-edcc-42dd-a976-019883307b6b-cert\") pod \"odh-model-controller-7998c67dc4-fkrdd\" (UID: \"516b1e47-edcc-42dd-a976-019883307b6b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:34.941878 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:34.941797 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsz52\" (UniqueName: \"kubernetes.io/projected/516b1e47-edcc-42dd-a976-019883307b6b-kube-api-access-tsz52\") pod \"odh-model-controller-7998c67dc4-fkrdd\" (UID: \"516b1e47-edcc-42dd-a976-019883307b6b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:35.003547 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.003515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:35.042234 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.042163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516b1e47-edcc-42dd-a976-019883307b6b-cert\") pod \"odh-model-controller-7998c67dc4-fkrdd\" (UID: \"516b1e47-edcc-42dd-a976-019883307b6b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:35.042234 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.042226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsz52\" (UniqueName: \"kubernetes.io/projected/516b1e47-edcc-42dd-a976-019883307b6b-kube-api-access-tsz52\") pod \"odh-model-controller-7998c67dc4-fkrdd\" (UID: \"516b1e47-edcc-42dd-a976-019883307b6b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:35.044798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.044774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/516b1e47-edcc-42dd-a976-019883307b6b-cert\") pod \"odh-model-controller-7998c67dc4-fkrdd\" (UID: \"516b1e47-edcc-42dd-a976-019883307b6b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:35.050630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.050605 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsz52\" (UniqueName: \"kubernetes.io/projected/516b1e47-edcc-42dd-a976-019883307b6b-kube-api-access-tsz52\") pod \"odh-model-controller-7998c67dc4-fkrdd\" (UID: \"516b1e47-edcc-42dd-a976-019883307b6b\") " pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:35.086100 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.086065 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:35.096961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.096740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:35.225102 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.225075 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fkrdd"] Jul 21 21:06:35.227428 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:35.227400 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516b1e47_edcc_42dd_a976_019883307b6b.slice/crio-1fbbc84a9b6729be9787f0b503d1bed71d92762280af59b00e710d20cdd03d46 WatchSource:0}: Error finding container 1fbbc84a9b6729be9787f0b503d1bed71d92762280af59b00e710d20cdd03d46: Status 404 returned error can't find the container with id 1fbbc84a9b6729be9787f0b503d1bed71d92762280af59b00e710d20cdd03d46 Jul 21 21:06:35.495633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.495597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:35.509683 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.509653 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ss4kl"] Jul 21 21:06:35.513000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.512978 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.524613 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.524592 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5w42m\"" Jul 21 21:06:35.528003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.527983 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 21:06:35.547098 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.547070 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ss4kl"] Jul 21 21:06:35.647521 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.647491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1239b8-a55a-462c-a1ab-8c0a241c6ac6-tls-certs\") pod \"model-serving-api-54d89bd5fc-ss4kl\" (UID: \"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6\") " pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.647696 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.647627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6t5\" (UniqueName: \"kubernetes.io/projected/0d1239b8-a55a-462c-a1ab-8c0a241c6ac6-kube-api-access-bp6t5\") pod \"model-serving-api-54d89bd5fc-ss4kl\" (UID: \"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6\") " pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.679266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.679234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:35.748688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.748593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6t5\" (UniqueName: \"kubernetes.io/projected/0d1239b8-a55a-462c-a1ab-8c0a241c6ac6-kube-api-access-bp6t5\") pod \"model-serving-api-54d89bd5fc-ss4kl\" (UID: \"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6\") " pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.748688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.748639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1239b8-a55a-462c-a1ab-8c0a241c6ac6-tls-certs\") pod \"model-serving-api-54d89bd5fc-ss4kl\" (UID: \"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6\") " pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.751176 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.751148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1239b8-a55a-462c-a1ab-8c0a241c6ac6-tls-certs\") pod \"model-serving-api-54d89bd5fc-ss4kl\" (UID: \"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6\") " pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.756524 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.756504 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6t5\" (UniqueName: \"kubernetes.io/projected/0d1239b8-a55a-462c-a1ab-8c0a241c6ac6-kube-api-access-bp6t5\") pod \"model-serving-api-54d89bd5fc-ss4kl\" (UID: \"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6\") " pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.824066 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.824026 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:35.993283 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:35.993067 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ss4kl"] Jul 21 21:06:35.996622 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:35.995623 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1239b8_a55a_462c_a1ab_8c0a241c6ac6.slice/crio-6026baa2eb9278f333963f75462561f8457eae933d59cb3b6a0d00bee08dea06 WatchSource:0}: Error finding container 6026baa2eb9278f333963f75462561f8457eae933d59cb3b6a0d00bee08dea06: Status 404 returned error can't find the container with id 6026baa2eb9278f333963f75462561f8457eae933d59cb3b6a0d00bee08dea06 Jul 21 21:06:36.115353 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:36.115302 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" event={"ID":"516b1e47-edcc-42dd-a976-019883307b6b","Type":"ContainerStarted","Data":"1fbbc84a9b6729be9787f0b503d1bed71d92762280af59b00e710d20cdd03d46"} Jul 21 21:06:36.116921 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:36.116886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" event={"ID":"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6","Type":"ContainerStarted","Data":"6026baa2eb9278f333963f75462561f8457eae933d59cb3b6a0d00bee08dea06"} Jul 21 21:06:39.141031 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:39.140931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" event={"ID":"0d1239b8-a55a-462c-a1ab-8c0a241c6ac6","Type":"ContainerStarted","Data":"63bc0a03290022b2e6873c08f84795a2e5e488ac433c24b78f32b877dca7687b"} Jul 21 21:06:39.141504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:39.141032 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:39.142789 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:39.142766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" event={"ID":"516b1e47-edcc-42dd-a976-019883307b6b","Type":"ContainerStarted","Data":"4f059d26cc759766186949dc4cf51fe5577eced7c4a7b59ceb884be386b799a5"} Jul 21 21:06:39.142905 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:39.142886 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:39.161773 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:39.161725 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" podStartSLOduration=1.451365754 podStartE2EDuration="4.161712003s" totalImagesPullingTime="2.710346249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:39.160423545 +0000 UTC m=+376.105044630" watchObservedRunningTime="2026-07-21 21:06:39.161712003 +0000 UTC m=+376.106333067" Jul 21 21:06:39.179930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:39.179880 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" podStartSLOduration=2.212837983 podStartE2EDuration="5.1798677s" totalImagesPullingTime="2.967029717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:39.178711764 +0000 UTC m=+376.123332831" watchObservedRunningTime="2026-07-21 21:06:39.1798677 +0000 UTC m=+376.124488764" Jul 21 21:06:44.682231 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.682180 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:44.736513 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.736482 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7"] Jul 21 21:06:44.739970 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.739943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:44.742742 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.742717 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 21:06:44.742867 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.742746 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dsrzf\"" Jul 21 21:06:44.751265 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.751243 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84"] Jul 21 21:06:44.754324 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.754305 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7"] Jul 21 21:06:44.754440 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.754423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:44.761506 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.761459 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zkznx\"" Jul 21 21:06:44.761622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.761501 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 21:06:44.775542 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.775504 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84"] Jul 21 21:06:44.778965 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.778939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:44.927731 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.927699 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2t2\" (UniqueName: \"kubernetes.io/projected/bf1921af-bb51-43c6-b544-cecff337f092-kube-api-access-tc2t2\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:44.927914 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.927760 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:44.927914 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.927805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf1921af-bb51-43c6-b544-cecff337f092-cert\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:44.927914 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.927835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq7q\" (UniqueName: \"kubernetes.io/projected/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-kube-api-access-6fq7q\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:44.976727 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:44.976648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:45.028991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.028957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2t2\" (UniqueName: \"kubernetes.io/projected/bf1921af-bb51-43c6-b544-cecff337f092-kube-api-access-tc2t2\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:45.029154 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.029016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:45.029154 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.029048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf1921af-bb51-43c6-b544-cecff337f092-cert\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:45.029154 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.029073 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq7q\" (UniqueName: \"kubernetes.io/projected/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-kube-api-access-6fq7q\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:45.029269 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:06:45.029182 2570 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 21 21:06:45.029269 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:06:45.029202 2570 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 21 21:06:45.029269 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:06:45.029247 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-cert podName:de3fe3cc-480a-44ab-bf75-c0a5d46e2118 nodeName:}" failed. No retries permitted until 2026-07-21 21:06:45.529229126 +0000 UTC m=+382.473850174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-cert") pod "llmisvc-controller-manager-777c6bd7d5-r2v84" (UID: "de3fe3cc-480a-44ab-bf75-c0a5d46e2118") : secret "llmisvc-webhook-server-cert" not found Jul 21 21:06:45.029269 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:06:45.029264 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1921af-bb51-43c6-b544-cecff337f092-cert podName:bf1921af-bb51-43c6-b544-cecff337f092 nodeName:}" failed. No retries permitted until 2026-07-21 21:06:45.529255889 +0000 UTC m=+382.473876934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bf1921af-bb51-43c6-b544-cecff337f092-cert") pod "kserve-controller-manager-d6f4b65b8-pvvx7" (UID: "bf1921af-bb51-43c6-b544-cecff337f092") : secret "kserve-webhook-server-cert" not found Jul 21 21:06:45.051216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.051182 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq7q\" (UniqueName: \"kubernetes.io/projected/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-kube-api-access-6fq7q\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:45.051216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.051182 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2t2\" (UniqueName: \"kubernetes.io/projected/bf1921af-bb51-43c6-b544-cecff337f092-kube-api-access-tc2t2\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:45.102909 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.102868 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b46444884-9dqzl"] Jul 21 21:06:45.106469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.106444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.109551 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.109527 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 21:06:45.109679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.109558 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 21:06:45.110038 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.110020 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-twlhs\"" Jul 21 21:06:45.110641 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.110619 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 21:06:45.110752 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.110659 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 21:06:45.120424 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.120396 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b46444884-9dqzl"] Jul 21 21:06:45.162475 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.162439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:45.230771 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.230676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwg2w\" (UniqueName: \"kubernetes.io/projected/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-kube-api-access-bwg2w\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.230771 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.230729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-tmp\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.231007 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.230810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-tls-certs\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.256617 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.256586 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:45.320926 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.320894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:45.332004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.331972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-tmp\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.332138 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.332034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-tls-certs\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.332200 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.332137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwg2w\" (UniqueName: \"kubernetes.io/projected/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-kube-api-access-bwg2w\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.334403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.334356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-tmp\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.334654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.334632 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-tls-certs\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.341631 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.341606 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwg2w\" (UniqueName: \"kubernetes.io/projected/8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb-kube-api-access-bwg2w\") pod \"kube-auth-proxy-b46444884-9dqzl\" (UID: \"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb\") " pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.416677 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.416645 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" Jul 21 21:06:45.533977 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.533947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:45.533977 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.533987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf1921af-bb51-43c6-b544-cecff337f092-cert\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:45.536412 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.536389 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de3fe3cc-480a-44ab-bf75-c0a5d46e2118-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2v84\" (UID: \"de3fe3cc-480a-44ab-bf75-c0a5d46e2118\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:45.536534 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.536439 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf1921af-bb51-43c6-b544-cecff337f092-cert\") pod \"kserve-controller-manager-d6f4b65b8-pvvx7\" (UID: \"bf1921af-bb51-43c6-b544-cecff337f092\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:45.569449 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.569418 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b46444884-9dqzl"] Jul 21 21:06:45.587226 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:45.587201 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5a1071_4ed8_4a2a_8f8b_1196b6f276bb.slice/crio-cb5033be17bacbd17528cf9ea59aab7ddce68a2951a833fd7e8e3c78e0e12e7b WatchSource:0}: Error finding container cb5033be17bacbd17528cf9ea59aab7ddce68a2951a833fd7e8e3c78e0e12e7b: Status 404 returned error can't find the container with id cb5033be17bacbd17528cf9ea59aab7ddce68a2951a833fd7e8e3c78e0e12e7b Jul 21 21:06:45.651602 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.651577 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:45.667912 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.667883 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:45.798297 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.798265 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7"] Jul 21 21:06:45.799964 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:45.799929 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1921af_bb51_43c6_b544_cecff337f092.slice/crio-321d556ada3b0d9fb8a40d09567fbdf9d09c56627b4137594b22a0d13b7d5306 WatchSource:0}: Error finding container 321d556ada3b0d9fb8a40d09567fbdf9d09c56627b4137594b22a0d13b7d5306: Status 404 returned error can't find the container with id 321d556ada3b0d9fb8a40d09567fbdf9d09c56627b4137594b22a0d13b7d5306 Jul 21 21:06:45.817735 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:45.817709 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84"] Jul 21 21:06:45.818759 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:06:45.818737 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podde3fe3cc_480a_44ab_bf75_c0a5d46e2118.slice/crio-f5139c99698fac7f4e7f857a36d0420ffdf79e1fe88653ab95eb278c34c31342 WatchSource:0}: Error finding container f5139c99698fac7f4e7f857a36d0420ffdf79e1fe88653ab95eb278c34c31342: Status 404 returned error can't find the container with id f5139c99698fac7f4e7f857a36d0420ffdf79e1fe88653ab95eb278c34c31342 Jul 21 21:06:46.001436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.001395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:46.134683 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.134592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:46.175635 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.175593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" event={"ID":"de3fe3cc-480a-44ab-bf75-c0a5d46e2118","Type":"ContainerStarted","Data":"f5139c99698fac7f4e7f857a36d0420ffdf79e1fe88653ab95eb278c34c31342"} Jul 21 21:06:46.177044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.177012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" event={"ID":"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb","Type":"ContainerStarted","Data":"cb5033be17bacbd17528cf9ea59aab7ddce68a2951a833fd7e8e3c78e0e12e7b"} Jul 21 21:06:46.178379 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.178337 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" event={"ID":"bf1921af-bb51-43c6-b544-cecff337f092","Type":"ContainerStarted","Data":"321d556ada3b0d9fb8a40d09567fbdf9d09c56627b4137594b22a0d13b7d5306"} Jul 21 21:06:46.234753 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.234720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:46.929272 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.929192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:46.934539 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.932938 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk"] Jul 21 21:06:46.938625 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.938600 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:46.950202 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.948798 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 21:06:46.955270 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.954318 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8n7n4\"" Jul 21 21:06:46.955270 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.954521 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 21:06:46.955550 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.955532 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 21:06:46.955906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.955873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk"] Jul 21 21:06:46.957138 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:46.956919 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.048468 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.048402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49z62\" (UniqueName: \"kubernetes.io/projected/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-kube-api-access-49z62\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.062441 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.062260 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.149786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.149853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.149889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.149918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.149952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.149999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.150206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.150034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49z62\" (UniqueName: \"kubernetes.io/projected/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-kube-api-access-49z62\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.151306 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.151039 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.154146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.154101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.155722 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.155696 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.158210 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.158179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.158590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.158565 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.171698 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.171668 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.173794 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.173753 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49z62\" (UniqueName: \"kubernetes.io/projected/e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6-kube-api-access-49z62\") pod \"istiod-openshift-gateway-94bb8fbfd-clcpk\" (UID: \"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.257263 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.257188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:47.511381 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:47.511243 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk"] Jul 21 21:06:48.194416 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:48.194378 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" event={"ID":"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6","Type":"ContainerStarted","Data":"87e93e305a6b97cf3948471286acf5c9fe656bf2cb2654244539daba0aff70b1"} Jul 21 21:06:48.263289 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:48.263258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:50.149342 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.149249 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fkrdd" Jul 21 21:06:50.151804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.151777 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ss4kl" Jul 21 21:06:50.211875 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.211835 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" event={"ID":"de3fe3cc-480a-44ab-bf75-c0a5d46e2118","Type":"ContainerStarted","Data":"52ee6c875f727f75168c4ff2560c60de9edc76736c6f36f75c816c21034ede52"} Jul 21 21:06:50.212719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.212698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:06:50.214540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.214508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" event={"ID":"8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb","Type":"ContainerStarted","Data":"ac193554041e9f63dc28677ef8a43ce4de88f5bd0d260794059d97da08a80872"} Jul 21 21:06:50.219110 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.219078 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" event={"ID":"bf1921af-bb51-43c6-b544-cecff337f092","Type":"ContainerStarted","Data":"2b265f255c7bb0bbdbbbfbda9be55022c19d3807028ea49b2d90d1d2cb98d864"} Jul 21 21:06:50.219581 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.219558 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:06:50.235606 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.235480 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" podStartSLOduration=2.353014811 podStartE2EDuration="6.235461613s" totalImagesPullingTime="3.882446802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:50.231476674 +0000 UTC m=+387.176097740" watchObservedRunningTime="2026-07-21 21:06:50.235461613 +0000 UTC m=+387.180082679" Jul 21 21:06:50.250152 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.250092 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" podStartSLOduration=2.35385034 podStartE2EDuration="6.250072325s" totalImagesPullingTime="3.896221985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:50.248730584 +0000 UTC m=+387.193351653" watchObservedRunningTime="2026-07-21 21:06:50.250072325 +0000 UTC m=+387.194693393" Jul 21 21:06:50.268434 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:50.268385 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b46444884-9dqzl" podStartSLOduration=1.162656061 podStartE2EDuration="5.268346943s" totalImagesPullingTime="4.105690882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:50.267076893 +0000 UTC m=+387.211697975" watchObservedRunningTime="2026-07-21 21:06:50.268346943 +0000 UTC m=+387.212968010" Jul 21 21:06:51.182304 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:51.182267 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:06:51.182662 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:51.182337 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 21:06:52.141717 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.141673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:52.208046 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.208005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:52.231225 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.231187 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" event={"ID":"e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6","Type":"ContainerStarted","Data":"e02afa6042b12f86c4eaa3a25b563e6f2587ba38ce0791a1ae1c2845eccbb301"} Jul 21 21:06:52.231733 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.231563 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:06:52.233415 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.233388 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-clcpk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 21:06:52.233507 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.233445 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" podUID="e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:06:52.265003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.264946 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" podStartSLOduration=2.605720904 podStartE2EDuration="6.264930111s" totalImagesPullingTime="3.659209207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:52.26383274 +0000 UTC m=+389.208453821" watchObservedRunningTime="2026-07-21 21:06:52.264930111 +0000 UTC m=+389.209551176" Jul 21 21:06:52.973900 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:52.973861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:53.101024 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:53.100984 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:53.203665 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:53.203629 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:06:53.236841 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:06:53.236741 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-clcpk" Jul 21 21:07:22.233869 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:22.233836 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2v84" Jul 21 21:07:22.237011 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:22.236990 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pvvx7" Jul 21 21:07:35.622755 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:35.622721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:07:35.719220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:35.719189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:07:35.796442 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:35.796415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:07:50.939181 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.939098 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2"] Jul 21 21:07:50.942065 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.942046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:50.945126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.945106 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:07:50.945241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.945120 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:07:50.945241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.945147 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lwr2p\"" Jul 21 21:07:50.950959 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.950936 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2"] Jul 21 21:07:50.997960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.997925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:50.998121 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.998068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:50.998121 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:50.998098 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blsh4\" (UniqueName: \"kubernetes.io/projected/db02354b-33ea-41bc-9522-7623a1fdded8-kube-api-access-blsh4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.099665 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.099625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.099861 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.099670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blsh4\" (UniqueName: \"kubernetes.io/projected/db02354b-33ea-41bc-9522-7623a1fdded8-kube-api-access-blsh4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.099861 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.099711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.100050 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.100028 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.100050 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.100049 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.108465 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.108434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blsh4\" (UniqueName: \"kubernetes.io/projected/db02354b-33ea-41bc-9522-7623a1fdded8-kube-api-access-blsh4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.251678 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.251598 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:51.386631 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.386603 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2"] Jul 21 21:07:51.388927 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:07:51.388882 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb02354b_33ea_41bc_9522_7623a1fdded8.slice/crio-f867de68ecb4594992a030429fccb926558ebf491fe04d94181bcafd337258ce WatchSource:0}: Error finding container f867de68ecb4594992a030429fccb926558ebf491fe04d94181bcafd337258ce: Status 404 returned error can't find the container with id f867de68ecb4594992a030429fccb926558ebf491fe04d94181bcafd337258ce Jul 21 21:07:51.513227 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.513148 2570 generic.go:356] "Generic (PLEG): container finished" podID="db02354b-33ea-41bc-9522-7623a1fdded8" containerID="769c44b015bad22e8e8bad75f66b0c24bd4dff4fac86ddb8b778b4014caafd2a" exitCode=0 Jul 21 21:07:51.513378 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.513226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" event={"ID":"db02354b-33ea-41bc-9522-7623a1fdded8","Type":"ContainerDied","Data":"769c44b015bad22e8e8bad75f66b0c24bd4dff4fac86ddb8b778b4014caafd2a"} Jul 21 21:07:51.513378 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.513281 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" event={"ID":"db02354b-33ea-41bc-9522-7623a1fdded8","Type":"ContainerStarted","Data":"f867de68ecb4594992a030429fccb926558ebf491fe04d94181bcafd337258ce"} Jul 21 21:07:51.730046 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.730008 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm"] Jul 21 21:07:51.732445 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.732422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.742074 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.742048 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm"] Jul 21 21:07:51.805593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.805507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.805593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.805559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.805779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.805646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmc6n\" (UniqueName: \"kubernetes.io/projected/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-kube-api-access-dmc6n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.906596 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.906535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc6n\" (UniqueName: \"kubernetes.io/projected/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-kube-api-access-dmc6n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.906793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.906629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.906793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.906664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.907021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.907006 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.907057 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.907027 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:51.915027 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:51.914999 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc6n\" (UniqueName: \"kubernetes.io/projected/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-kube-api-access-dmc6n\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:52.041941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.041907 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:52.133961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.133934 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x"] Jul 21 21:07:52.136633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.136615 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.145671 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.145636 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x"] Jul 21 21:07:52.172830 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.172808 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm"] Jul 21 21:07:52.174921 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:07:52.174893 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fbd1b0_6f1f_4da7_b882_78c89fc83c07.slice/crio-750187d45cc83202b188d7546147b5c7bed22c3e3817d869fccc88b648d78370 WatchSource:0}: Error finding container 750187d45cc83202b188d7546147b5c7bed22c3e3817d869fccc88b648d78370: Status 404 returned error can't find the container with id 750187d45cc83202b188d7546147b5c7bed22c3e3817d869fccc88b648d78370 Jul 21 21:07:52.209023 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.208995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.209112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.209076 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdsgp\" (UniqueName: \"kubernetes.io/projected/3e950e96-708d-4bec-a54e-38b88d6d03f4-kube-api-access-wdsgp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.209173 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.209150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.309748 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.309717 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.309918 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.309793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdsgp\" (UniqueName: \"kubernetes.io/projected/3e950e96-708d-4bec-a54e-38b88d6d03f4-kube-api-access-wdsgp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.309918 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.309817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.310112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.310089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.310180 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.310134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.317997 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.317977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdsgp\" (UniqueName: \"kubernetes.io/projected/3e950e96-708d-4bec-a54e-38b88d6d03f4-kube-api-access-wdsgp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.447499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.447468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:52.526608 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.526570 2570 generic.go:356] "Generic (PLEG): container finished" podID="db02354b-33ea-41bc-9522-7623a1fdded8" containerID="069d59e429c7d1c64c0ed497bafcb134c547635705c5c0852773dcebeb2cce68" exitCode=0 Jul 21 21:07:52.526792 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.526653 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" event={"ID":"db02354b-33ea-41bc-9522-7623a1fdded8","Type":"ContainerDied","Data":"069d59e429c7d1c64c0ed497bafcb134c547635705c5c0852773dcebeb2cce68"} Jul 21 21:07:52.528356 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.528327 2570 generic.go:356] "Generic (PLEG): container finished" podID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerID="29bc80682302cd6d976670721e69d660b929c59ea36328aa577c5c38b022819d" exitCode=0 Jul 21 21:07:52.528471 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.528404 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" event={"ID":"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07","Type":"ContainerDied","Data":"29bc80682302cd6d976670721e69d660b929c59ea36328aa577c5c38b022819d"} Jul 21 21:07:52.528471 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.528463 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" event={"ID":"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07","Type":"ContainerStarted","Data":"750187d45cc83202b188d7546147b5c7bed22c3e3817d869fccc88b648d78370"} Jul 21 21:07:52.535921 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.535898 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7"] Jul 21 21:07:52.539764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.539744 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.552610 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.552583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7"] Jul 21 21:07:52.593081 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.593058 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x"] Jul 21 21:07:52.595132 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:07:52.595105 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e950e96_708d_4bec_a54e_38b88d6d03f4.slice/crio-34cb3c91a968ae828ea99f9d8e52f248f44082d57a76bfc7e70a2eb3d5711a2f WatchSource:0}: Error finding container 34cb3c91a968ae828ea99f9d8e52f248f44082d57a76bfc7e70a2eb3d5711a2f: Status 404 returned error can't find the container with id 34cb3c91a968ae828ea99f9d8e52f248f44082d57a76bfc7e70a2eb3d5711a2f Jul 21 21:07:52.612315 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.612282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sj8\" (UniqueName: \"kubernetes.io/projected/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-kube-api-access-d4sj8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.612445 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.612384 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.612445 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.612409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.713757 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.713730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sj8\" (UniqueName: \"kubernetes.io/projected/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-kube-api-access-d4sj8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.713891 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.713783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.713891 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.713814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.714081 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.714062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.714118 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.714102 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.722181 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.722155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sj8\" (UniqueName: \"kubernetes.io/projected/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-kube-api-access-d4sj8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.871446 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.871411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:52.998236 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:52.998214 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7"] Jul 21 21:07:52.999889 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:07:52.999866 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b00904_c24d_4ae8_ac42_1fe8995d8a32.slice/crio-fdeaf4abf05278da38ccc648856bd894d0959287c104e408c6fca75e905cb5ad WatchSource:0}: Error finding container fdeaf4abf05278da38ccc648856bd894d0959287c104e408c6fca75e905cb5ad: Status 404 returned error can't find the container with id fdeaf4abf05278da38ccc648856bd894d0959287c104e408c6fca75e905cb5ad Jul 21 21:07:53.534650 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.534613 2570 generic.go:356] "Generic (PLEG): container finished" podID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerID="0da9763bc412fba4668bd0e0418e89763c4254343dd8a8493ffc7fa25f2e27b3" exitCode=0 Jul 21 21:07:53.535123 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.534696 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" event={"ID":"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07","Type":"ContainerDied","Data":"0da9763bc412fba4668bd0e0418e89763c4254343dd8a8493ffc7fa25f2e27b3"} Jul 21 21:07:53.536267 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.536190 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerID="6c9b017a2690f34aa94b78b1bf658936e940417ce53b4d2c5fa2a3cf33efd394" exitCode=0 Jul 21 21:07:53.536267 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.536242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" event={"ID":"3e950e96-708d-4bec-a54e-38b88d6d03f4","Type":"ContainerDied","Data":"6c9b017a2690f34aa94b78b1bf658936e940417ce53b4d2c5fa2a3cf33efd394"} Jul 21 21:07:53.536403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.536293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" event={"ID":"3e950e96-708d-4bec-a54e-38b88d6d03f4","Type":"ContainerStarted","Data":"34cb3c91a968ae828ea99f9d8e52f248f44082d57a76bfc7e70a2eb3d5711a2f"} Jul 21 21:07:53.538935 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.538915 2570 generic.go:356] "Generic (PLEG): container finished" podID="db02354b-33ea-41bc-9522-7623a1fdded8" containerID="d503aeb79dc591e30fa9098d9fc7ed902755d5c4b25a1862d7e1f3f209a6c537" exitCode=0 Jul 21 21:07:53.539016 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.538993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" event={"ID":"db02354b-33ea-41bc-9522-7623a1fdded8","Type":"ContainerDied","Data":"d503aeb79dc591e30fa9098d9fc7ed902755d5c4b25a1862d7e1f3f209a6c537"} Jul 21 21:07:53.540739 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.540692 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerID="bc5f1d21446493eab60a2a9e591a86047630c253cba4dc872d63f712657237e4" exitCode=0 Jul 21 21:07:53.540739 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.540719 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" event={"ID":"e9b00904-c24d-4ae8-ac42-1fe8995d8a32","Type":"ContainerDied","Data":"bc5f1d21446493eab60a2a9e591a86047630c253cba4dc872d63f712657237e4"} Jul 21 21:07:53.540927 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:53.540763 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" event={"ID":"e9b00904-c24d-4ae8-ac42-1fe8995d8a32","Type":"ContainerStarted","Data":"fdeaf4abf05278da38ccc648856bd894d0959287c104e408c6fca75e905cb5ad"} Jul 21 21:07:54.553261 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.553171 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerID="8a075bed3d48472b3504ef672b87b2e1ed91221d41b4d4a5b633e7b973e525fe" exitCode=0 Jul 21 21:07:54.553749 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.553255 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" event={"ID":"3e950e96-708d-4bec-a54e-38b88d6d03f4","Type":"ContainerDied","Data":"8a075bed3d48472b3504ef672b87b2e1ed91221d41b4d4a5b633e7b973e525fe"} Jul 21 21:07:54.555151 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.555130 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerID="11b5e12cfc83a64a55c319279e0f4748e441cb416a98be8c33da49aa08d75aae" exitCode=0 Jul 21 21:07:54.555231 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.555209 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" event={"ID":"e9b00904-c24d-4ae8-ac42-1fe8995d8a32","Type":"ContainerDied","Data":"11b5e12cfc83a64a55c319279e0f4748e441cb416a98be8c33da49aa08d75aae"} Jul 21 21:07:54.557996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.557972 2570 generic.go:356] "Generic (PLEG): container finished" podID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerID="1253f6ba4100368b7ecefc1016fe7ca33a9cf6e51796f54517f2e516683d2e93" exitCode=0 Jul 21 21:07:54.558079 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.558042 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" event={"ID":"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07","Type":"ContainerDied","Data":"1253f6ba4100368b7ecefc1016fe7ca33a9cf6e51796f54517f2e516683d2e93"} Jul 21 21:07:54.690323 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.690299 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:54.731664 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.731640 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blsh4\" (UniqueName: \"kubernetes.io/projected/db02354b-33ea-41bc-9522-7623a1fdded8-kube-api-access-blsh4\") pod \"db02354b-33ea-41bc-9522-7623a1fdded8\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " Jul 21 21:07:54.731778 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.731687 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-bundle\") pod \"db02354b-33ea-41bc-9522-7623a1fdded8\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " Jul 21 21:07:54.731778 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.731762 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-util\") pod \"db02354b-33ea-41bc-9522-7623a1fdded8\" (UID: \"db02354b-33ea-41bc-9522-7623a1fdded8\") " Jul 21 21:07:54.732294 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.732262 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-bundle" (OuterVolumeSpecName: "bundle") pod "db02354b-33ea-41bc-9522-7623a1fdded8" (UID: "db02354b-33ea-41bc-9522-7623a1fdded8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:54.733855 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.733832 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db02354b-33ea-41bc-9522-7623a1fdded8-kube-api-access-blsh4" (OuterVolumeSpecName: "kube-api-access-blsh4") pod "db02354b-33ea-41bc-9522-7623a1fdded8" (UID: "db02354b-33ea-41bc-9522-7623a1fdded8"). InnerVolumeSpecName "kube-api-access-blsh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:54.741188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.741163 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-util" (OuterVolumeSpecName: "util") pod "db02354b-33ea-41bc-9522-7623a1fdded8" (UID: "db02354b-33ea-41bc-9522-7623a1fdded8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:54.833134 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.833053 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blsh4\" (UniqueName: \"kubernetes.io/projected/db02354b-33ea-41bc-9522-7623a1fdded8-kube-api-access-blsh4\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:54.833134 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.833085 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:54.833134 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:54.833098 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db02354b-33ea-41bc-9522-7623a1fdded8-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:55.566451 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.566422 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerID="2566549029d4338d720b2164fc0ca0a6473519cb4a17737e88fec54c487602c8" exitCode=0 Jul 21 21:07:55.566780 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.566503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" event={"ID":"3e950e96-708d-4bec-a54e-38b88d6d03f4","Type":"ContainerDied","Data":"2566549029d4338d720b2164fc0ca0a6473519cb4a17737e88fec54c487602c8"} Jul 21 21:07:55.568338 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.568312 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" event={"ID":"db02354b-33ea-41bc-9522-7623a1fdded8","Type":"ContainerDied","Data":"f867de68ecb4594992a030429fccb926558ebf491fe04d94181bcafd337258ce"} Jul 21 21:07:55.568455 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.568319 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2" Jul 21 21:07:55.568455 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.568396 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f867de68ecb4594992a030429fccb926558ebf491fe04d94181bcafd337258ce" Jul 21 21:07:55.570525 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.570503 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerID="bbfd25c4edba4635d68ceac402576bce8369dac01f23977219ef21db406126f7" exitCode=0 Jul 21 21:07:55.570602 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.570591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" event={"ID":"e9b00904-c24d-4ae8-ac42-1fe8995d8a32","Type":"ContainerDied","Data":"bbfd25c4edba4635d68ceac402576bce8369dac01f23977219ef21db406126f7"} Jul 21 21:07:55.699252 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.699228 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:55.739469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.739439 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmc6n\" (UniqueName: \"kubernetes.io/projected/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-kube-api-access-dmc6n\") pod \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " Jul 21 21:07:55.739622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.739540 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-bundle\") pod \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " Jul 21 21:07:55.739622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.739589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-util\") pod \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\" (UID: \"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07\") " Jul 21 21:07:55.740472 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.740440 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-bundle" (OuterVolumeSpecName: "bundle") pod "a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" (UID: "a0fbd1b0-6f1f-4da7-b882-78c89fc83c07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:55.741599 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.741572 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-kube-api-access-dmc6n" (OuterVolumeSpecName: "kube-api-access-dmc6n") pod "a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" (UID: "a0fbd1b0-6f1f-4da7-b882-78c89fc83c07"). InnerVolumeSpecName "kube-api-access-dmc6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:55.745261 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.745235 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-util" (OuterVolumeSpecName: "util") pod "a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" (UID: "a0fbd1b0-6f1f-4da7-b882-78c89fc83c07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:55.840614 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.840587 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmc6n\" (UniqueName: \"kubernetes.io/projected/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-kube-api-access-dmc6n\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:55.840614 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.840611 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:55.840614 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:55.840619 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0fbd1b0-6f1f-4da7-b882-78c89fc83c07-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:56.577535 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.577504 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" Jul 21 21:07:56.577922 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.577531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm" event={"ID":"a0fbd1b0-6f1f-4da7-b882-78c89fc83c07","Type":"ContainerDied","Data":"750187d45cc83202b188d7546147b5c7bed22c3e3817d869fccc88b648d78370"} Jul 21 21:07:56.577922 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.577579 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750187d45cc83202b188d7546147b5c7bed22c3e3817d869fccc88b648d78370" Jul 21 21:07:56.714812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.714786 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:56.747552 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.747522 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:56.748802 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.748777 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-util\") pod \"3e950e96-708d-4bec-a54e-38b88d6d03f4\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " Jul 21 21:07:56.748854 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.748813 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-bundle\") pod \"3e950e96-708d-4bec-a54e-38b88d6d03f4\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " Jul 21 21:07:56.748894 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.748861 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdsgp\" (UniqueName: \"kubernetes.io/projected/3e950e96-708d-4bec-a54e-38b88d6d03f4-kube-api-access-wdsgp\") pod \"3e950e96-708d-4bec-a54e-38b88d6d03f4\" (UID: \"3e950e96-708d-4bec-a54e-38b88d6d03f4\") " Jul 21 21:07:56.749495 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.749462 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-bundle" (OuterVolumeSpecName: "bundle") pod "3e950e96-708d-4bec-a54e-38b88d6d03f4" (UID: "3e950e96-708d-4bec-a54e-38b88d6d03f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:56.751272 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.751244 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e950e96-708d-4bec-a54e-38b88d6d03f4-kube-api-access-wdsgp" (OuterVolumeSpecName: "kube-api-access-wdsgp") pod "3e950e96-708d-4bec-a54e-38b88d6d03f4" (UID: "3e950e96-708d-4bec-a54e-38b88d6d03f4"). InnerVolumeSpecName "kube-api-access-wdsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:56.754759 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.754736 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-util" (OuterVolumeSpecName: "util") pod "3e950e96-708d-4bec-a54e-38b88d6d03f4" (UID: "3e950e96-708d-4bec-a54e-38b88d6d03f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:56.849929 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.849844 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-bundle\") pod \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " Jul 21 21:07:56.849929 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.849906 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-util\") pod \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " Jul 21 21:07:56.850105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.849936 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4sj8\" (UniqueName: \"kubernetes.io/projected/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-kube-api-access-d4sj8\") pod \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\" (UID: \"e9b00904-c24d-4ae8-ac42-1fe8995d8a32\") " Jul 21 21:07:56.850105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.850103 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:56.850170 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.850112 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e950e96-708d-4bec-a54e-38b88d6d03f4-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:56.850170 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.850121 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdsgp\" (UniqueName: \"kubernetes.io/projected/3e950e96-708d-4bec-a54e-38b88d6d03f4-kube-api-access-wdsgp\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:56.850464 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.850438 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-bundle" (OuterVolumeSpecName: "bundle") pod "e9b00904-c24d-4ae8-ac42-1fe8995d8a32" (UID: "e9b00904-c24d-4ae8-ac42-1fe8995d8a32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:56.852174 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.852155 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-kube-api-access-d4sj8" (OuterVolumeSpecName: "kube-api-access-d4sj8") pod "e9b00904-c24d-4ae8-ac42-1fe8995d8a32" (UID: "e9b00904-c24d-4ae8-ac42-1fe8995d8a32"). InnerVolumeSpecName "kube-api-access-d4sj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:56.855147 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.855124 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-util" (OuterVolumeSpecName: "util") pod "e9b00904-c24d-4ae8-ac42-1fe8995d8a32" (UID: "e9b00904-c24d-4ae8-ac42-1fe8995d8a32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:56.950963 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.950924 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-util\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:56.950963 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.950956 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4sj8\" (UniqueName: \"kubernetes.io/projected/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-kube-api-access-d4sj8\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:56.950963 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:56.950967 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9b00904-c24d-4ae8-ac42-1fe8995d8a32-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:07:57.584085 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:57.584054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" event={"ID":"3e950e96-708d-4bec-a54e-38b88d6d03f4","Type":"ContainerDied","Data":"34cb3c91a968ae828ea99f9d8e52f248f44082d57a76bfc7e70a2eb3d5711a2f"} Jul 21 21:07:57.584436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:57.584103 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34cb3c91a968ae828ea99f9d8e52f248f44082d57a76bfc7e70a2eb3d5711a2f" Jul 21 21:07:57.584436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:57.584066 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x" Jul 21 21:07:57.586165 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:57.586133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" event={"ID":"e9b00904-c24d-4ae8-ac42-1fe8995d8a32","Type":"ContainerDied","Data":"fdeaf4abf05278da38ccc648856bd894d0959287c104e408c6fca75e905cb5ad"} Jul 21 21:07:57.586165 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:57.586157 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7" Jul 21 21:07:57.586341 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:57.586181 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdeaf4abf05278da38ccc648856bd894d0959287c104e408c6fca75e905cb5ad" Jul 21 21:07:58.833651 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:07:58.833623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:02.234209 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:02.234174 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:04.217923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:04.217892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:05.526335 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526299 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk"] Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526650 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="util" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526663 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="util" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526673 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="pull" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526679 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="pull" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526687 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="util" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526694 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="util" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526700 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="extract" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526706 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="extract" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526713 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="util" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526719 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="util" Jul 21 21:08:05.526721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526728 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="util" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526732 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="util" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526738 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="pull" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526743 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="pull" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526748 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="pull" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526753 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="pull" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526761 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="pull" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526765 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="pull" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526772 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526776 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526780 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526785 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526791 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526795 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526843 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b00904-c24d-4ae8-ac42-1fe8995d8a32" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526851 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e950e96-708d-4bec-a54e-38b88d6d03f4" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526858 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="db02354b-33ea-41bc-9522-7623a1fdded8" containerName="extract" Jul 21 21:08:05.527054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.526863 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0fbd1b0-6f1f-4da7-b882-78c89fc83c07" containerName="extract" Jul 21 21:08:05.529779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.529762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:05.532893 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.532870 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:08:05.533983 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.533965 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nhwrv\"" Jul 21 21:08:05.534089 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.533966 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:08:05.540513 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.540488 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk"] Jul 21 21:08:05.624639 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.624606 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwtv\" (UniqueName: \"kubernetes.io/projected/b74a276e-c269-43ad-99a9-d4d9f0a135bc-kube-api-access-9rwtv\") pod \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" (UID: \"b74a276e-c269-43ad-99a9-d4d9f0a135bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:05.725682 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.725650 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwtv\" (UniqueName: \"kubernetes.io/projected/b74a276e-c269-43ad-99a9-d4d9f0a135bc-kube-api-access-9rwtv\") pod \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" (UID: \"b74a276e-c269-43ad-99a9-d4d9f0a135bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:05.737207 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.737179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwtv\" (UniqueName: \"kubernetes.io/projected/b74a276e-c269-43ad-99a9-d4d9f0a135bc-kube-api-access-9rwtv\") pod \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" (UID: \"b74a276e-c269-43ad-99a9-d4d9f0a135bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:05.793840 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.793762 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:05.840496 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.840464 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:05.975047 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:05.975023 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk"] Jul 21 21:08:05.976629 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:08:05.976602 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74a276e_c269_43ad_99a9_d4d9f0a135bc.slice/crio-f15cb2b324a516d40d710096f663caf92f0df96db793eb0e8ea5064965af4b9e WatchSource:0}: Error finding container f15cb2b324a516d40d710096f663caf92f0df96db793eb0e8ea5064965af4b9e: Status 404 returned error can't find the container with id f15cb2b324a516d40d710096f663caf92f0df96db793eb0e8ea5064965af4b9e Jul 21 21:08:06.198342 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:06.198309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:06.635729 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:06.635654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" event={"ID":"b74a276e-c269-43ad-99a9-d4d9f0a135bc","Type":"ContainerStarted","Data":"f15cb2b324a516d40d710096f663caf92f0df96db793eb0e8ea5064965af4b9e"} Jul 21 21:08:07.632825 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:07.632777 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:08.647725 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:08.647686 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" event={"ID":"b74a276e-c269-43ad-99a9-d4d9f0a135bc","Type":"ContainerStarted","Data":"d9ab712c5c43a87f8e67f036da03dfaf7af775d9eaaaaa8dade71ad7d6b60239"} Jul 21 21:08:08.648167 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:08.647781 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:08.665511 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:08.665453 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" podStartSLOduration=1.8994023960000002 podStartE2EDuration="3.665439641s" totalImagesPullingTime="1.766037245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:08.664754566 +0000 UTC m=+465.609375630" watchObservedRunningTime="2026-07-21 21:08:08.665439641 +0000 UTC m=+465.610060705" Jul 21 21:08:09.608750 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:09.608717 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:10.636856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:10.636824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:10.734421 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:10.734388 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:12.404107 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.404072 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n"] Jul 21 21:08:12.407673 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.407651 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:12.410388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.410348 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 21:08:12.410477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.410396 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8zwwd\"" Jul 21 21:08:12.417983 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.417963 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n"] Jul 21 21:08:12.484988 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.484955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2xj\" (UniqueName: \"kubernetes.io/projected/2b43ab62-d073-495e-841e-f085c544f1c1-kube-api-access-9q2xj\") pod \"dns-operator-controller-manager-864dd78c4c-srz8n\" (UID: \"2b43ab62-d073-495e-841e-f085c544f1c1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:12.500567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.500542 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:12.585513 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.585482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2xj\" (UniqueName: \"kubernetes.io/projected/2b43ab62-d073-495e-841e-f085c544f1c1-kube-api-access-9q2xj\") pod \"dns-operator-controller-manager-864dd78c4c-srz8n\" (UID: \"2b43ab62-d073-495e-841e-f085c544f1c1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:12.594051 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.594032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2xj\" (UniqueName: \"kubernetes.io/projected/2b43ab62-d073-495e-841e-f085c544f1c1-kube-api-access-9q2xj\") pod \"dns-operator-controller-manager-864dd78c4c-srz8n\" (UID: \"2b43ab62-d073-495e-841e-f085c544f1c1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:12.717648 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.717617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:12.854541 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:12.854517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n"] Jul 21 21:08:12.856576 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:08:12.856544 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b43ab62_d073_495e_841e_f085c544f1c1.slice/crio-651ac56f8e97a33fc8bba53c14fb2e97f38040364b7712e6f35e7d406b120d85 WatchSource:0}: Error finding container 651ac56f8e97a33fc8bba53c14fb2e97f38040364b7712e6f35e7d406b120d85: Status 404 returned error can't find the container with id 651ac56f8e97a33fc8bba53c14fb2e97f38040364b7712e6f35e7d406b120d85 Jul 21 21:08:13.053869 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:13.053784 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:13.275949 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:13.275918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:13.673518 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:13.673486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" event={"ID":"2b43ab62-d073-495e-841e-f085c544f1c1","Type":"ContainerStarted","Data":"651ac56f8e97a33fc8bba53c14fb2e97f38040364b7712e6f35e7d406b120d85"} Jul 21 21:08:15.395482 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:15.395392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:15.471801 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:15.471758 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:15.687625 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:15.687582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" event={"ID":"2b43ab62-d073-495e-841e-f085c544f1c1","Type":"ContainerStarted","Data":"bc1ec73b7d81e1eeb714f9870525862bd89ed21cb594b62d528881a0e8d71be8"} Jul 21 21:08:15.687811 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:15.687657 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:15.707834 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:15.707785 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" podStartSLOduration=1.534297163 podStartE2EDuration="3.707772251s" totalImagesPullingTime="2.173475088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:15.705541708 +0000 UTC m=+472.650162790" watchObservedRunningTime="2026-07-21 21:08:15.707772251 +0000 UTC m=+472.652393315" Jul 21 21:08:19.655186 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:19.655158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:22.555594 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.555563 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk"] Jul 21 21:08:22.556069 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.555754 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" containerName="manager" containerID="cri-o://d9ab712c5c43a87f8e67f036da03dfaf7af775d9eaaaaa8dade71ad7d6b60239" gracePeriod=2 Jul 21 21:08:22.574326 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.574296 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk"] Jul 21 21:08:22.578588 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:08:22.578554 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b74a276e-c269-43ad-99a9-d4d9f0a135bc\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-hpbmk\": pods \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:22.580994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.580969 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5"] Jul 21 21:08:22.581315 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.581303 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" containerName="manager" Jul 21 21:08:22.581402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.581316 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" containerName="manager" Jul 21 21:08:22.581466 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.581453 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" containerName="manager" Jul 21 21:08:22.584518 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.584497 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:22.596699 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.596674 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5"] Jul 21 21:08:22.610775 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:08:22.610746 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:22.673062 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.673042 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbz6\" (UniqueName: \"kubernetes.io/projected/d7388aa8-9190-4187-82c6-a6b4d5005b50-kube-api-access-npbz6\") pod \"limitador-operator-controller-manager-859d7b78bd-55bt5\" (UID: \"d7388aa8-9190-4187-82c6-a6b4d5005b50\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:22.723801 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.723768 2570 generic.go:356] "Generic (PLEG): container finished" podID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" containerID="d9ab712c5c43a87f8e67f036da03dfaf7af775d9eaaaaa8dade71ad7d6b60239" exitCode=0 Jul 21 21:08:22.774317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.774289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbz6\" (UniqueName: \"kubernetes.io/projected/d7388aa8-9190-4187-82c6-a6b4d5005b50-kube-api-access-npbz6\") pod \"limitador-operator-controller-manager-859d7b78bd-55bt5\" (UID: \"d7388aa8-9190-4187-82c6-a6b4d5005b50\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:22.783449 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.783427 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbz6\" (UniqueName: \"kubernetes.io/projected/d7388aa8-9190-4187-82c6-a6b4d5005b50-kube-api-access-npbz6\") pod \"limitador-operator-controller-manager-859d7b78bd-55bt5\" (UID: \"d7388aa8-9190-4187-82c6-a6b4d5005b50\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:22.801520 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.801495 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:22.803983 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:08:22.803960 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:22.946163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.946135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:22.975929 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.975898 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rwtv\" (UniqueName: \"kubernetes.io/projected/b74a276e-c269-43ad-99a9-d4d9f0a135bc-kube-api-access-9rwtv\") pod \"b74a276e-c269-43ad-99a9-d4d9f0a135bc\" (UID: \"b74a276e-c269-43ad-99a9-d4d9f0a135bc\") " Jul 21 21:08:22.977990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:22.977964 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74a276e-c269-43ad-99a9-d4d9f0a135bc-kube-api-access-9rwtv" (OuterVolumeSpecName: "kube-api-access-9rwtv") pod "b74a276e-c269-43ad-99a9-d4d9f0a135bc" (UID: "b74a276e-c269-43ad-99a9-d4d9f0a135bc"). InnerVolumeSpecName "kube-api-access-9rwtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:08:23.076950 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.076919 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwtv\" (UniqueName: \"kubernetes.io/projected/b74a276e-c269-43ad-99a9-d4d9f0a135bc-kube-api-access-9rwtv\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:08:23.091486 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.091463 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5"] Jul 21 21:08:23.092744 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:08:23.092717 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7388aa8_9190_4187_82c6_a6b4d5005b50.slice/crio-77d18e0374eae1a41b0834bcb2f904b72abfe6971bcdfc5b1f887596aa3b9f93 WatchSource:0}: Error finding container 77d18e0374eae1a41b0834bcb2f904b72abfe6971bcdfc5b1f887596aa3b9f93: Status 404 returned error can't find the container with id 77d18e0374eae1a41b0834bcb2f904b72abfe6971bcdfc5b1f887596aa3b9f93 Jul 21 21:08:23.556655 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:08:23.556624 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hpbmk\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:23.557206 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.557185 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74a276e-c269-43ad-99a9-d4d9f0a135bc" path="/var/lib/kubelet/pods/b74a276e-c269-43ad-99a9-d4d9f0a135bc/volumes" Jul 21 21:08:23.653071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.653043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:23.730477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.730444 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hpbmk" Jul 21 21:08:23.730629 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.730483 2570 scope.go:117] "RemoveContainer" containerID="d9ab712c5c43a87f8e67f036da03dfaf7af775d9eaaaaa8dade71ad7d6b60239" Jul 21 21:08:23.732693 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.732659 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" event={"ID":"d7388aa8-9190-4187-82c6-a6b4d5005b50","Type":"ContainerStarted","Data":"9df6c024af2b86a8158f2953e93ddd964ba6ee477d4cbcec630bbef39540f867"} Jul 21 21:08:23.732846 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.732726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" event={"ID":"d7388aa8-9190-4187-82c6-a6b4d5005b50","Type":"ContainerStarted","Data":"77d18e0374eae1a41b0834bcb2f904b72abfe6971bcdfc5b1f887596aa3b9f93"} Jul 21 21:08:23.732913 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.732880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:23.743090 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.743069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:23.752873 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:23.752835 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" podStartSLOduration=1.752821382 podStartE2EDuration="1.752821382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:23.751031015 +0000 UTC m=+480.695652095" watchObservedRunningTime="2026-07-21 21:08:23.752821382 +0000 UTC m=+480.697442446" Jul 21 21:08:26.695042 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:26.695013 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-srz8n" Jul 21 21:08:28.399133 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:28.399103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:28.499303 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:28.499270 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:28.990930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:28.990898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:29.056376 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:29.056327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:32.795703 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:32.795661 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:32.895800 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:32.895765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:34.740793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:34.740756 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-55bt5" Jul 21 21:08:38.202129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:38.202072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:38.302715 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:38.302678 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:46.404657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:46.404620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:46.503482 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:46.503442 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:51.226611 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:51.226579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:51.319992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:51.319947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:51.499399 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:51.499290 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:52.851602 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.851566 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:08:52.857853 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.857822 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:52.860738 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.860715 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 21:08:52.860861 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.860751 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lwr2p\"" Jul 21 21:08:52.865539 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.865502 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:08:52.879769 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.879741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:52.898008 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.897978 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/97c6069a-4b7f-46b4-9c7d-85ec165c2207-config-file\") pod \"limitador-limitador-64c94fd64f-p28wn\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:52.898137 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.898057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8m2j\" (UniqueName: \"kubernetes.io/projected/97c6069a-4b7f-46b4-9c7d-85ec165c2207-kube-api-access-w8m2j\") pod \"limitador-limitador-64c94fd64f-p28wn\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:52.947326 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.947271 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:08:52.954924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.954898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:52.999001 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.998966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8m2j\" (UniqueName: \"kubernetes.io/projected/97c6069a-4b7f-46b4-9c7d-85ec165c2207-kube-api-access-w8m2j\") pod \"limitador-limitador-64c94fd64f-p28wn\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:52.999167 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.999037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/97c6069a-4b7f-46b4-9c7d-85ec165c2207-config-file\") pod \"limitador-limitador-64c94fd64f-p28wn\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:52.999632 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:52.999614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/97c6069a-4b7f-46b4-9c7d-85ec165c2207-config-file\") pod \"limitador-limitador-64c94fd64f-p28wn\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:53.007805 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.007780 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8m2j\" (UniqueName: \"kubernetes.io/projected/97c6069a-4b7f-46b4-9c7d-85ec165c2207-kube-api-access-w8m2j\") pod \"limitador-limitador-64c94fd64f-p28wn\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:53.028992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.028966 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:53.170510 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.170479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:53.307477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.307451 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:08:53.665708 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.665677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:53.764798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.764748 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:53.891188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:53.891159 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" event={"ID":"97c6069a-4b7f-46b4-9c7d-85ec165c2207","Type":"ContainerStarted","Data":"b97d4799d73f22470e4599aa4ff66f2690ee58367a07a5781241be14c531353a"} Jul 21 21:08:56.910951 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:56.910913 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" event={"ID":"97c6069a-4b7f-46b4-9c7d-85ec165c2207","Type":"ContainerStarted","Data":"dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117"} Jul 21 21:08:56.911402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:56.911050 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:08:56.928507 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:56.928456 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" podStartSLOduration=2.253162705 podStartE2EDuration="4.928442983s" totalImagesPullingTime="2.675280278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:56.926297623 +0000 UTC m=+513.870918693" watchObservedRunningTime="2026-07-21 21:08:56.928442983 +0000 UTC m=+513.873064054" Jul 21 21:08:59.425186 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:59.425149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:08:59.541863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:08:59.541829 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:00.251912 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:00.251869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:00.352990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:00.352956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:01.117509 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:01.117478 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:01.174077 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:01.174044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:01.280675 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:01.280642 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:01.865982 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:01.865952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:01.969000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:01.968968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:07.420202 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:07.420171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:07.520898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:07.520868 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:07.917605 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:07.917573 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:09:10.990699 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:10.990667 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:09:10.991086 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:10.990857 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" podUID="97c6069a-4b7f-46b4-9c7d-85ec165c2207" containerName="limitador" containerID="cri-o://dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117" gracePeriod=30 Jul 21 21:09:11.535819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.535791 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:09:11.664382 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.664329 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8m2j\" (UniqueName: \"kubernetes.io/projected/97c6069a-4b7f-46b4-9c7d-85ec165c2207-kube-api-access-w8m2j\") pod \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " Jul 21 21:09:11.664554 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.664427 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/97c6069a-4b7f-46b4-9c7d-85ec165c2207-config-file\") pod \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\" (UID: \"97c6069a-4b7f-46b4-9c7d-85ec165c2207\") " Jul 21 21:09:11.664815 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.664775 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c6069a-4b7f-46b4-9c7d-85ec165c2207-config-file" (OuterVolumeSpecName: "config-file") pod "97c6069a-4b7f-46b4-9c7d-85ec165c2207" (UID: "97c6069a-4b7f-46b4-9c7d-85ec165c2207"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:09:11.666842 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.666818 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c6069a-4b7f-46b4-9c7d-85ec165c2207-kube-api-access-w8m2j" (OuterVolumeSpecName: "kube-api-access-w8m2j") pod "97c6069a-4b7f-46b4-9c7d-85ec165c2207" (UID: "97c6069a-4b7f-46b4-9c7d-85ec165c2207"). InnerVolumeSpecName "kube-api-access-w8m2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:09:11.765220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.765195 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8m2j\" (UniqueName: \"kubernetes.io/projected/97c6069a-4b7f-46b4-9c7d-85ec165c2207-kube-api-access-w8m2j\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:11.765220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.765222 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/97c6069a-4b7f-46b4-9c7d-85ec165c2207-config-file\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:11.973402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.973317 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85fdf46566-6v6w9"] Jul 21 21:09:11.998998 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.998970 2570 generic.go:356] "Generic (PLEG): container finished" podID="97c6069a-4b7f-46b4-9c7d-85ec165c2207" containerID="dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117" exitCode=0 Jul 21 21:09:11.999454 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.999023 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" Jul 21 21:09:11.999454 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.999058 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" event={"ID":"97c6069a-4b7f-46b4-9c7d-85ec165c2207","Type":"ContainerDied","Data":"dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117"} Jul 21 21:09:11.999454 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.999109 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-p28wn" event={"ID":"97c6069a-4b7f-46b4-9c7d-85ec165c2207","Type":"ContainerDied","Data":"b97d4799d73f22470e4599aa4ff66f2690ee58367a07a5781241be14c531353a"} Jul 21 21:09:11.999454 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:11.999137 2570 scope.go:117] "RemoveContainer" containerID="dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117" Jul 21 21:09:12.008580 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.008562 2570 scope.go:117] "RemoveContainer" containerID="dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117" Jul 21 21:09:12.008857 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:09:12.008837 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117\": container with ID starting with dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117 not found: ID does not exist" containerID="dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117" Jul 21 21:09:12.008930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.008869 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117"} err="failed to get container status \"dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117\": rpc error: code = NotFound desc = could not find container \"dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117\": container with ID starting with dfeca2f4039518a6efdc4fcb358b380722cf5fefec37c358ef5ed8ec686e3117 not found: ID does not exist" Jul 21 21:09:12.021850 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.021822 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:09:12.026985 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.026964 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-p28wn"] Jul 21 21:09:12.034680 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.034653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:12.101825 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.101796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:12.201700 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.201668 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:12.268159 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:12.268083 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:13.556047 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:13.556007 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c6069a-4b7f-46b4-9c7d-85ec165c2207" path="/var/lib/kubelet/pods/97c6069a-4b7f-46b4-9c7d-85ec165c2207/volumes" Jul 21 21:09:14.215525 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.215489 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:14.312856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.312823 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:14.697739 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.697708 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b"] Jul 21 21:09:14.698299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.698082 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c6069a-4b7f-46b4-9c7d-85ec165c2207" containerName="limitador" Jul 21 21:09:14.698299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.698093 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c6069a-4b7f-46b4-9c7d-85ec165c2207" containerName="limitador" Jul 21 21:09:14.698299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.698221 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c6069a-4b7f-46b4-9c7d-85ec165c2207" containerName="limitador" Jul 21 21:09:14.702599 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.702575 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:14.705662 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.705637 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 21:09:14.705779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.705637 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 21:09:14.705779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.705647 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:09:14.706875 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.706854 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ngb4m\"" Jul 21 21:09:14.712600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.712564 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b"] Jul 21 21:09:14.734960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.734919 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:14.789476 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.789451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9d1351aa-977d-4ee9-a50b-ca7a5db6da56-data\") pod \"postgres-69fb4594cf-jpj6b\" (UID: \"9d1351aa-977d-4ee9-a50b-ca7a5db6da56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:14.789584 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.789479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9d24\" (UniqueName: \"kubernetes.io/projected/9d1351aa-977d-4ee9-a50b-ca7a5db6da56-kube-api-access-z9d24\") pod \"postgres-69fb4594cf-jpj6b\" (UID: \"9d1351aa-977d-4ee9-a50b-ca7a5db6da56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:14.890028 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.889996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9d1351aa-977d-4ee9-a50b-ca7a5db6da56-data\") pod \"postgres-69fb4594cf-jpj6b\" (UID: \"9d1351aa-977d-4ee9-a50b-ca7a5db6da56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:14.890028 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.890028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9d24\" (UniqueName: \"kubernetes.io/projected/9d1351aa-977d-4ee9-a50b-ca7a5db6da56-kube-api-access-z9d24\") pod \"postgres-69fb4594cf-jpj6b\" (UID: \"9d1351aa-977d-4ee9-a50b-ca7a5db6da56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:14.890418 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.890396 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9d1351aa-977d-4ee9-a50b-ca7a5db6da56-data\") pod \"postgres-69fb4594cf-jpj6b\" (UID: \"9d1351aa-977d-4ee9-a50b-ca7a5db6da56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:14.898937 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:14.898913 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9d24\" (UniqueName: \"kubernetes.io/projected/9d1351aa-977d-4ee9-a50b-ca7a5db6da56-kube-api-access-z9d24\") pod \"postgres-69fb4594cf-jpj6b\" (UID: \"9d1351aa-977d-4ee9-a50b-ca7a5db6da56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:15.015005 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:15.014943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:15.148862 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:15.148836 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b"] Jul 21 21:09:15.150392 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:09:15.150339 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1351aa_977d_4ee9_a50b_ca7a5db6da56.slice/crio-92d8814dab8252d030cd5cbb2714baf9397924ee3449eef8de139a8c893a37ec WatchSource:0}: Error finding container 92d8814dab8252d030cd5cbb2714baf9397924ee3449eef8de139a8c893a37ec: Status 404 returned error can't find the container with id 92d8814dab8252d030cd5cbb2714baf9397924ee3449eef8de139a8c893a37ec Jul 21 21:09:15.428992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:15.428949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:15.490877 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:15.490843 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:16.024836 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:16.024788 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" event={"ID":"9d1351aa-977d-4ee9-a50b-ca7a5db6da56","Type":"ContainerStarted","Data":"92d8814dab8252d030cd5cbb2714baf9397924ee3449eef8de139a8c893a37ec"} Jul 21 21:09:21.054600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:21.054564 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" event={"ID":"9d1351aa-977d-4ee9-a50b-ca7a5db6da56","Type":"ContainerStarted","Data":"c956e83b051d90f9bc9c82067c4484d95461748dc5968281d756c55be59148f3"} Jul 21 21:09:21.054973 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:21.054639 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:21.080449 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:21.080402 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" podStartSLOduration=2.189158458 podStartE2EDuration="7.080386182s" totalImagesPullingTime="4.891227724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:09:21.079053936 +0000 UTC m=+538.023675001" watchObservedRunningTime="2026-07-21 21:09:21.080386182 +0000 UTC m=+538.025007244" Jul 21 21:09:27.090711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:27.090681 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jpj6b" Jul 21 21:09:27.434377 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:27.434317 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:27.625181 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:27.625149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:33.839640 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.839602 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:33.844128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.844102 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-689bb4f647-dmkpz"] Jul 21 21:09:33.865524 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.865498 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-689bb4f647-dmkpz"] Jul 21 21:09:33.865659 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.865609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:33.868266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.868238 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 21:09:33.868413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.868324 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qmmzs\"" Jul 21 21:09:33.868413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.868387 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 21:09:33.918814 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.918782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:33.952019 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.951993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-webhook-cert\") pod \"maas-controller-689bb4f647-dmkpz\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:33.952097 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.952053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphgc\" (UniqueName: \"kubernetes.io/projected/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-kube-api-access-cphgc\") pod \"maas-controller-689bb4f647-dmkpz\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:33.978259 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:33.978234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:34.023077 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.023050 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55d6556f9b-flxpc"] Jul 21 21:09:34.025868 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.025850 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.035188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.035158 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-flxpc"] Jul 21 21:09:34.053015 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.052993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-webhook-cert\") pod \"maas-controller-689bb4f647-dmkpz\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:34.053115 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.053044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphgc\" (UniqueName: \"kubernetes.io/projected/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-kube-api-access-cphgc\") pod \"maas-controller-689bb4f647-dmkpz\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:34.055587 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.055569 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-webhook-cert\") pod \"maas-controller-689bb4f647-dmkpz\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:34.062000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.061976 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphgc\" (UniqueName: \"kubernetes.io/projected/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-kube-api-access-cphgc\") pod \"maas-controller-689bb4f647-dmkpz\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:34.067903 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.067874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:34.154099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.154033 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a900aa40-d271-4389-9213-8c04fed84e1a-webhook-cert\") pod \"maas-controller-55d6556f9b-flxpc\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.154211 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.154097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8rx\" (UniqueName: \"kubernetes.io/projected/a900aa40-d271-4389-9213-8c04fed84e1a-kube-api-access-bc8rx\") pod \"maas-controller-55d6556f9b-flxpc\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.177117 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.177089 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:34.254868 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.254830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a900aa40-d271-4389-9213-8c04fed84e1a-webhook-cert\") pod \"maas-controller-55d6556f9b-flxpc\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.255004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.254898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8rx\" (UniqueName: \"kubernetes.io/projected/a900aa40-d271-4389-9213-8c04fed84e1a-kube-api-access-bc8rx\") pod \"maas-controller-55d6556f9b-flxpc\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.257657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.257415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a900aa40-d271-4389-9213-8c04fed84e1a-webhook-cert\") pod \"maas-controller-55d6556f9b-flxpc\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.265131 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.265101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8rx\" (UniqueName: \"kubernetes.io/projected/a900aa40-d271-4389-9213-8c04fed84e1a-kube-api-access-bc8rx\") pod \"maas-controller-55d6556f9b-flxpc\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.308835 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.308809 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-689bb4f647-dmkpz"] Jul 21 21:09:34.311260 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:09:34.311234 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc460f8f3_bc31_40f1_96f5_36f1e2d6cbe7.slice/crio-b1efb28a8541bcf8a59e2b3bd3de03889912e9a7abfc88f827d79bb49fda8db1 WatchSource:0}: Error finding container b1efb28a8541bcf8a59e2b3bd3de03889912e9a7abfc88f827d79bb49fda8db1: Status 404 returned error can't find the container with id b1efb28a8541bcf8a59e2b3bd3de03889912e9a7abfc88f827d79bb49fda8db1 Jul 21 21:09:34.337689 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.337664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:34.471137 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:34.471107 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-flxpc"] Jul 21 21:09:34.472250 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:09:34.472221 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda900aa40_d271_4389_9213_8c04fed84e1a.slice/crio-4311ce2fc78c68d98ee425430b3b66c0c2a385ade1667ac6f6e4cf712dc11d43 WatchSource:0}: Error finding container 4311ce2fc78c68d98ee425430b3b66c0c2a385ade1667ac6f6e4cf712dc11d43: Status 404 returned error can't find the container with id 4311ce2fc78c68d98ee425430b3b66c0c2a385ade1667ac6f6e4cf712dc11d43 Jul 21 21:09:35.134644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:35.134604 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689bb4f647-dmkpz" event={"ID":"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7","Type":"ContainerStarted","Data":"b1efb28a8541bcf8a59e2b3bd3de03889912e9a7abfc88f827d79bb49fda8db1"} Jul 21 21:09:35.136682 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:35.136653 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-flxpc" event={"ID":"a900aa40-d271-4389-9213-8c04fed84e1a","Type":"ContainerStarted","Data":"4311ce2fc78c68d98ee425430b3b66c0c2a385ade1667ac6f6e4cf712dc11d43"} Jul 21 21:09:36.875958 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:36.875920 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:36.974442 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:36.974323 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:09:36.995665 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:36.995597 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85fdf46566-6v6w9" podUID="3a3036da-ca81-406c-bfb8-eb5f15030631" containerName="console" containerID="cri-o://ed20bea09f24fd1d04aeda7b32bcc4358a0a8aae0786024a9f38d3ed9f0249ad" gracePeriod=15 Jul 21 21:09:37.152308 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.152277 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a3036da-ca81-406c-bfb8-eb5f15030631" containerID="ed20bea09f24fd1d04aeda7b32bcc4358a0a8aae0786024a9f38d3ed9f0249ad" exitCode=0 Jul 21 21:09:37.152507 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.152380 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fdf46566-6v6w9" event={"ID":"3a3036da-ca81-406c-bfb8-eb5f15030631","Type":"ContainerDied","Data":"ed20bea09f24fd1d04aeda7b32bcc4358a0a8aae0786024a9f38d3ed9f0249ad"} Jul 21 21:09:37.154110 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.154082 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689bb4f647-dmkpz" event={"ID":"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7","Type":"ContainerStarted","Data":"f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7"} Jul 21 21:09:37.154262 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.154184 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:37.155647 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.155615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-flxpc" event={"ID":"a900aa40-d271-4389-9213-8c04fed84e1a","Type":"ContainerStarted","Data":"6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35"} Jul 21 21:09:37.155752 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.155681 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:37.175081 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.175032 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-689bb4f647-dmkpz" podStartSLOduration=1.7824300499999999 podStartE2EDuration="4.175014421s" totalImagesPullingTime="2.392584371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:09:37.172921455 +0000 UTC m=+554.117542521" watchObservedRunningTime="2026-07-21 21:09:37.175014421 +0000 UTC m=+554.119635486" Jul 21 21:09:37.194584 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.194542 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55d6556f9b-flxpc" podStartSLOduration=0.968588426 podStartE2EDuration="3.194529305s" totalImagesPullingTime="2.225940879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:09:37.192198898 +0000 UTC m=+554.136819980" watchObservedRunningTime="2026-07-21 21:09:37.194529305 +0000 UTC m=+554.139150370" Jul 21 21:09:37.255726 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.255695 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:09:37.288814 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.288773 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-serving-cert\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.288974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.288835 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-trusted-ca-bundle\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.288974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.288858 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-oauth-config\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.288974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.288890 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/3a3036da-ca81-406c-bfb8-eb5f15030631-kube-api-access-fwj9l\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.288974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.288926 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-service-ca\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.288974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.288953 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-oauth-serving-cert\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.289198 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.289028 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-console-config\") pod \"3a3036da-ca81-406c-bfb8-eb5f15030631\" (UID: \"3a3036da-ca81-406c-bfb8-eb5f15030631\") " Jul 21 21:09:37.290146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.289630 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:09:37.290146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.289638 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-console-config" (OuterVolumeSpecName: "console-config") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:09:37.290146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.290079 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-service-ca" (OuterVolumeSpecName: "service-ca") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:09:37.290417 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.290146 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:09:37.291637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.291596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3036da-ca81-406c-bfb8-eb5f15030631-kube-api-access-fwj9l" (OuterVolumeSpecName: "kube-api-access-fwj9l") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "kube-api-access-fwj9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:09:37.291840 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.291807 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:09:37.291981 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.291960 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3a3036da-ca81-406c-bfb8-eb5f15030631" (UID: "3a3036da-ca81-406c-bfb8-eb5f15030631"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:09:37.389651 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389616 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-trusted-ca-bundle\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:37.389651 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389649 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-oauth-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:37.389856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389662 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/3a3036da-ca81-406c-bfb8-eb5f15030631-kube-api-access-fwj9l\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:37.389856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389675 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:37.389856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389687 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-oauth-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:37.389856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389700 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a3036da-ca81-406c-bfb8-eb5f15030631-console-config\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:37.389856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:37.389711 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a3036da-ca81-406c-bfb8-eb5f15030631-console-serving-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:38.162584 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:38.162539 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fdf46566-6v6w9" event={"ID":"3a3036da-ca81-406c-bfb8-eb5f15030631","Type":"ContainerDied","Data":"6cc62f7c6a931095ae6ffce58fd744220d92cdf90723b085446f984c0c8c5831"} Jul 21 21:09:38.162584 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:38.162567 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fdf46566-6v6w9" Jul 21 21:09:38.163073 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:38.162612 2570 scope.go:117] "RemoveContainer" containerID="ed20bea09f24fd1d04aeda7b32bcc4358a0a8aae0786024a9f38d3ed9f0249ad" Jul 21 21:09:38.182229 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:38.182202 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85fdf46566-6v6w9"] Jul 21 21:09:38.188690 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:38.188667 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85fdf46566-6v6w9"] Jul 21 21:09:39.556342 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:39.556307 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3036da-ca81-406c-bfb8-eb5f15030631" path="/var/lib/kubelet/pods/3a3036da-ca81-406c-bfb8-eb5f15030631/volumes" Jul 21 21:09:48.165021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.164986 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:09:48.165429 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.165060 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:48.224345 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.224317 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-689bb4f647-dmkpz"] Jul 21 21:09:48.224604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.224553 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-689bb4f647-dmkpz" podUID="c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" containerName="manager" containerID="cri-o://f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7" gracePeriod=30 Jul 21 21:09:48.475477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.475445 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:48.576772 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.576739 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cphgc\" (UniqueName: \"kubernetes.io/projected/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-kube-api-access-cphgc\") pod \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " Jul 21 21:09:48.576961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.576791 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-webhook-cert\") pod \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\" (UID: \"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7\") " Jul 21 21:09:48.579178 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.579144 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" (UID: "c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:09:48.579289 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.579180 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-kube-api-access-cphgc" (OuterVolumeSpecName: "kube-api-access-cphgc") pod "c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" (UID: "c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7"). InnerVolumeSpecName "kube-api-access-cphgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:09:48.677573 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.677533 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cphgc\" (UniqueName: \"kubernetes.io/projected/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-kube-api-access-cphgc\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:48.677573 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:48.677565 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7-webhook-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:09:49.222247 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.222210 2570 generic.go:356] "Generic (PLEG): container finished" podID="c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" containerID="f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7" exitCode=0 Jul 21 21:09:49.222719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.222272 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689bb4f647-dmkpz" Jul 21 21:09:49.222719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.222302 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689bb4f647-dmkpz" event={"ID":"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7","Type":"ContainerDied","Data":"f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7"} Jul 21 21:09:49.222719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.222388 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689bb4f647-dmkpz" event={"ID":"c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7","Type":"ContainerDied","Data":"b1efb28a8541bcf8a59e2b3bd3de03889912e9a7abfc88f827d79bb49fda8db1"} Jul 21 21:09:49.222719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.222426 2570 scope.go:117] "RemoveContainer" containerID="f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7" Jul 21 21:09:49.231934 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.231790 2570 scope.go:117] "RemoveContainer" containerID="f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7" Jul 21 21:09:49.232023 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:09:49.232006 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7\": container with ID starting with f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7 not found: ID does not exist" containerID="f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7" Jul 21 21:09:49.232066 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.232033 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7"} err="failed to get container status \"f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7\": rpc error: code = NotFound desc = could not find container \"f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7\": container with ID starting with f28abcd7a77688510996b456ea26d46a1cd4eb715d2cc57c8ceae62b962d2fc7 not found: ID does not exist" Jul 21 21:09:49.244877 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.244854 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-689bb4f647-dmkpz"] Jul 21 21:09:49.251213 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.251191 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-689bb4f647-dmkpz"] Jul 21 21:09:49.556724 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:09:49.556637 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" path="/var/lib/kubelet/pods/c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7/volumes" Jul 21 21:10:10.434622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:10.434577 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:10.575754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:10.575716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:10.886133 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:10.886036 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:11.044056 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044015 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv"] Jul 21 21:10:11.044407 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044393 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" containerName="manager" Jul 21 21:10:11.044460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044408 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" containerName="manager" Jul 21 21:10:11.044460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044432 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3036da-ca81-406c-bfb8-eb5f15030631" containerName="console" Jul 21 21:10:11.044460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044438 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3036da-ca81-406c-bfb8-eb5f15030631" containerName="console" Jul 21 21:10:11.044575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044493 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c460f8f3-bc31-40f1-96f5-36f1e2d6cbe7" containerName="manager" Jul 21 21:10:11.044575 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.044503 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3036da-ca81-406c-bfb8-eb5f15030631" containerName="console" Jul 21 21:10:11.052299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.052280 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.055529 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.055503 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 21:10:11.055776 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.055515 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h486b\"" Jul 21 21:10:11.057853 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.057831 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv"] Jul 21 21:10:11.171440 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.171412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494b6ca-a709-4c4d-af65-c026af2520e9-openshift-service-ca\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.171607 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.171455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltw4d\" (UniqueName: \"kubernetes.io/projected/e494b6ca-a709-4c4d-af65-c026af2520e9-kube-api-access-ltw4d\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.171607 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.171518 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494b6ca-a709-4c4d-af65-c026af2520e9-maas-api-tls\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.272737 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.272704 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494b6ca-a709-4c4d-af65-c026af2520e9-openshift-service-ca\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.272936 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.272746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltw4d\" (UniqueName: \"kubernetes.io/projected/e494b6ca-a709-4c4d-af65-c026af2520e9-kube-api-access-ltw4d\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.272936 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.272782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494b6ca-a709-4c4d-af65-c026af2520e9-maas-api-tls\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.273446 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.273422 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e494b6ca-a709-4c4d-af65-c026af2520e9-openshift-service-ca\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.275304 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.275282 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e494b6ca-a709-4c4d-af65-c026af2520e9-maas-api-tls\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.281515 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.281491 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltw4d\" (UniqueName: \"kubernetes.io/projected/e494b6ca-a709-4c4d-af65-c026af2520e9-kube-api-access-ltw4d\") pod \"maas-api-8dff7b7df-kprrv\" (UID: \"e494b6ca-a709-4c4d-af65-c026af2520e9\") " pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.289930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.289898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:11.363599 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.363570 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:11.420217 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.420176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:11.525775 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.525741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:11.531723 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.531691 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv"] Jul 21 21:10:11.533513 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:10:11.533485 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode494b6ca_a709_4c4d_af65_c026af2520e9.slice/crio-76efbcf6446f41e35a455518c0edb820a67ec23b51af07284e117025d855a390 WatchSource:0}: Error finding container 76efbcf6446f41e35a455518c0edb820a67ec23b51af07284e117025d855a390: Status 404 returned error can't find the container with id 76efbcf6446f41e35a455518c0edb820a67ec23b51af07284e117025d855a390 Jul 21 21:10:11.534141 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.534120 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hzfw4"] Jul 21 21:10:11.539711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.539694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.542329 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.542307 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r5v4s\"" Jul 21 21:10:11.542567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.542546 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 21:10:11.548864 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.548767 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hzfw4"] Jul 21 21:10:11.558330 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.558297 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-g5x69"] Jul 21 21:10:11.570816 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.563528 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.575733 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.575706 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-g5x69"] Jul 21 21:10:11.675694 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.675658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9-plugins-config-volume\") pod \"payload-processing-5bf47cd988-g5x69\" (UID: \"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9\") " pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.675860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.675711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdp7w\" (UniqueName: \"kubernetes.io/projected/54066cd0-970a-42c7-84a8-67bd17e03028-kube-api-access-qdp7w\") pod \"payload-pre-processing-7677b774bb-hzfw4\" (UID: \"54066cd0-970a-42c7-84a8-67bd17e03028\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.675860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.675739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54066cd0-970a-42c7-84a8-67bd17e03028-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hzfw4\" (UID: \"54066cd0-970a-42c7-84a8-67bd17e03028\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.675860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.675769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2bcq\" (UniqueName: \"kubernetes.io/projected/3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9-kube-api-access-w2bcq\") pod \"payload-processing-5bf47cd988-g5x69\" (UID: \"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9\") " pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.776943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.776860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9-plugins-config-volume\") pod \"payload-processing-5bf47cd988-g5x69\" (UID: \"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9\") " pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.776943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.776903 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdp7w\" (UniqueName: \"kubernetes.io/projected/54066cd0-970a-42c7-84a8-67bd17e03028-kube-api-access-qdp7w\") pod \"payload-pre-processing-7677b774bb-hzfw4\" (UID: \"54066cd0-970a-42c7-84a8-67bd17e03028\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.776943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.776922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54066cd0-970a-42c7-84a8-67bd17e03028-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hzfw4\" (UID: \"54066cd0-970a-42c7-84a8-67bd17e03028\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.776943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.776946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2bcq\" (UniqueName: \"kubernetes.io/projected/3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9-kube-api-access-w2bcq\") pod \"payload-processing-5bf47cd988-g5x69\" (UID: \"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9\") " pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.777624 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.777600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9-plugins-config-volume\") pod \"payload-processing-5bf47cd988-g5x69\" (UID: \"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9\") " pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.777743 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.777725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54066cd0-970a-42c7-84a8-67bd17e03028-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hzfw4\" (UID: \"54066cd0-970a-42c7-84a8-67bd17e03028\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.786655 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.786630 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdp7w\" (UniqueName: \"kubernetes.io/projected/54066cd0-970a-42c7-84a8-67bd17e03028-kube-api-access-qdp7w\") pod \"payload-pre-processing-7677b774bb-hzfw4\" (UID: \"54066cd0-970a-42c7-84a8-67bd17e03028\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.786742 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.786680 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2bcq\" (UniqueName: \"kubernetes.io/projected/3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9-kube-api-access-w2bcq\") pod \"payload-processing-5bf47cd988-g5x69\" (UID: \"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9\") " pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:11.852848 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.852816 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:11.880071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:11.880044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:12.004771 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.004741 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hzfw4"] Jul 21 21:10:12.006633 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:10:12.006604 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54066cd0_970a_42c7_84a8_67bd17e03028.slice/crio-53d6e92d0b9363b08261083de6a7e547dcef2c17abe79b0df0c0115e4438a04b WatchSource:0}: Error finding container 53d6e92d0b9363b08261083de6a7e547dcef2c17abe79b0df0c0115e4438a04b: Status 404 returned error can't find the container with id 53d6e92d0b9363b08261083de6a7e547dcef2c17abe79b0df0c0115e4438a04b Jul 21 21:10:12.036224 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.036200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-g5x69"] Jul 21 21:10:12.037639 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:10:12.037605 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbe19d1_4ed9_4b12_8e49_eff86da1bcb9.slice/crio-ade97d1bcd63cb3f12260865c1e1e11a1ce50713e5374d27b3bda01c4d7c565b WatchSource:0}: Error finding container ade97d1bcd63cb3f12260865c1e1e11a1ce50713e5374d27b3bda01c4d7c565b: Status 404 returned error can't find the container with id ade97d1bcd63cb3f12260865c1e1e11a1ce50713e5374d27b3bda01c4d7c565b Jul 21 21:10:12.239897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.239866 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:12.302617 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.302536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:12.351621 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.351581 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" event={"ID":"54066cd0-970a-42c7-84a8-67bd17e03028","Type":"ContainerStarted","Data":"53d6e92d0b9363b08261083de6a7e547dcef2c17abe79b0df0c0115e4438a04b"} Jul 21 21:10:12.352687 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.352665 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" event={"ID":"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9","Type":"ContainerStarted","Data":"ade97d1bcd63cb3f12260865c1e1e11a1ce50713e5374d27b3bda01c4d7c565b"} Jul 21 21:10:12.353865 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:12.353824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" event={"ID":"e494b6ca-a709-4c4d-af65-c026af2520e9","Type":"ContainerStarted","Data":"76efbcf6446f41e35a455518c0edb820a67ec23b51af07284e117025d855a390"} Jul 21 21:10:14.367887 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.367790 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" event={"ID":"3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9","Type":"ContainerStarted","Data":"e6141708a2d71f21394ee7d07ef11a0ac461023ed1b09034e3f0a22a7c092281"} Jul 21 21:10:14.368338 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.367908 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:14.369371 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.369341 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" event={"ID":"e494b6ca-a709-4c4d-af65-c026af2520e9","Type":"ContainerStarted","Data":"09ad677b7154404cccdf7361901fdfeae21ea2609a2b06474e797d7bf3a6f981"} Jul 21 21:10:14.369707 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.369419 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:14.370770 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.370749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" event={"ID":"54066cd0-970a-42c7-84a8-67bd17e03028","Type":"ContainerStarted","Data":"09d982c76ae51e2a2551259f23d2e6623d0b2c6625a33885f714836c7cdba464"} Jul 21 21:10:14.370876 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.370857 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:14.387583 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.387540 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" podStartSLOduration=1.345222467 podStartE2EDuration="3.387527079s" totalImagesPullingTime="2.042304612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:10:14.385092545 +0000 UTC m=+591.329713613" watchObservedRunningTime="2026-07-21 21:10:14.387527079 +0000 UTC m=+591.332148143" Jul 21 21:10:14.402532 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.402486 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" podStartSLOduration=1.33974061 podStartE2EDuration="3.40247238s" totalImagesPullingTime="2.06273177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:10:14.399841638 +0000 UTC m=+591.344462703" watchObservedRunningTime="2026-07-21 21:10:14.40247238 +0000 UTC m=+591.347093445" Jul 21 21:10:14.419242 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:14.419174 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" podStartSLOduration=1.337440075 podStartE2EDuration="3.419159824s" totalImagesPullingTime="2.081719749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:10:14.417230348 +0000 UTC m=+591.361851441" watchObservedRunningTime="2026-07-21 21:10:14.419159824 +0000 UTC m=+591.363780888" Jul 21 21:10:20.382912 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:20.382871 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8dff7b7df-kprrv" Jul 21 21:10:21.421207 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:21.421171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:21.529267 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:21.529232 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:25.379503 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:25.379470 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-g5x69" Jul 21 21:10:25.379925 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:25.379645 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-hzfw4" Jul 21 21:10:39.173227 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:39.173187 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:44.041698 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.041622 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl"] Jul 21 21:10:44.048019 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.046663 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.051645 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.051617 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t48v6\"" Jul 21 21:10:44.051779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.051657 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 21:10:44.051779 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.051688 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 21:10:44.051886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.051784 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 21:10:44.052030 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.052013 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810765f3-e481-4551-8bcb-b850d3198e52-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.052106 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.052066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.052153 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.052117 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.052199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.052150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.052199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.052192 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbl4\" (UniqueName: \"kubernetes.io/projected/810765f3-e481-4551-8bcb-b850d3198e52-kube-api-access-5xbl4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.052298 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.052224 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.056437 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.056398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl"] Jul 21 21:10:44.063379 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.063330 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:44.153128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbl4\" (UniqueName: \"kubernetes.io/projected/810765f3-e481-4551-8bcb-b850d3198e52-kube-api-access-5xbl4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810765f3-e481-4551-8bcb-b850d3198e52-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153255 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153317 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153677 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153653 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153743 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.153814 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.153775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.155630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.155609 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/810765f3-e481-4551-8bcb-b850d3198e52-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.155975 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.155959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/810765f3-e481-4551-8bcb-b850d3198e52-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.162005 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.161980 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbl4\" (UniqueName: \"kubernetes.io/projected/810765f3-e481-4551-8bcb-b850d3198e52-kube-api-access-5xbl4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl\" (UID: \"810765f3-e481-4551-8bcb-b850d3198e52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.361893 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.361807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:44.506291 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.506265 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl"] Jul 21 21:10:44.508738 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:10:44.508711 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810765f3_e481_4551_8bcb_b850d3198e52.slice/crio-8a0ec2b7668b86f393972f30df0e4aaf7d005fb5bbfabdd8b1bebf1ccacef598 WatchSource:0}: Error finding container 8a0ec2b7668b86f393972f30df0e4aaf7d005fb5bbfabdd8b1bebf1ccacef598: Status 404 returned error can't find the container with id 8a0ec2b7668b86f393972f30df0e4aaf7d005fb5bbfabdd8b1bebf1ccacef598 Jul 21 21:10:44.510148 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.510128 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:10:44.539129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:44.539100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" event={"ID":"810765f3-e481-4551-8bcb-b850d3198e52","Type":"ContainerStarted","Data":"8a0ec2b7668b86f393972f30df0e4aaf7d005fb5bbfabdd8b1bebf1ccacef598"} Jul 21 21:10:46.558570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:46.558531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" event={"ID":"810765f3-e481-4551-8bcb-b850d3198e52","Type":"ContainerStarted","Data":"d67f6fdbc5782474157e4efd71c088e1cb05cfbbd09493fcdf8143b613ecda80"} Jul 21 21:10:46.579282 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:46.579223 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" podStartSLOduration=1.418714901 podStartE2EDuration="2.579204158s" totalImagesPullingTime="1.160489257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:10:46.57772669 +0000 UTC m=+623.522347756" watchObservedRunningTime="2026-07-21 21:10:46.579204158 +0000 UTC m=+623.523825224" Jul 21 21:10:49.661930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:49.661892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:10:54.362439 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:54.362408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:54.362834 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:54.362464 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:54.375462 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:54.375437 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:54.617141 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:54.617055 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl" Jul 21 21:10:59.068606 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:10:59.068573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:11:04.478204 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:04.478172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:11:15.744821 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:15.744779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:11:19.138650 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.138616 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8"] Jul 21 21:11:19.146183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.146159 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.149213 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.149190 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 21:11:19.154299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.154016 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8"] Jul 21 21:11:19.177114 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.177084 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:11:19.261927 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.261896 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4skq\" (UniqueName: \"kubernetes.io/projected/b4b7293a-7545-4cfa-a9e3-74a10253a82d-kube-api-access-k4skq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.262087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.261981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.262087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.262017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.262087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.262066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b7293a-7545-4cfa-a9e3-74a10253a82d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.262215 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.262136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.262215 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.262192 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363284 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363284 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4skq\" (UniqueName: \"kubernetes.io/projected/b4b7293a-7545-4cfa-a9e3-74a10253a82d-kube-api-access-k4skq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b7293a-7545-4cfa-a9e3-74a10253a82d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.363954 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.363925 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.364187 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.364155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.364498 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.364303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.365718 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.365695 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b7293a-7545-4cfa-a9e3-74a10253a82d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.366027 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.366008 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b7293a-7545-4cfa-a9e3-74a10253a82d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.372237 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.372216 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4skq\" (UniqueName: \"kubernetes.io/projected/b4b7293a-7545-4cfa-a9e3-74a10253a82d-kube-api-access-k4skq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8\" (UID: \"b4b7293a-7545-4cfa-a9e3-74a10253a82d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.458741 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.458713 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:19.600840 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.600817 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8"] Jul 21 21:11:19.603479 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:11:19.603450 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b7293a_7545_4cfa_a9e3_74a10253a82d.slice/crio-23887b3f8e001c99e5f29e9f752e9dc05c45f531139d832a55c9e9c77584eca0 WatchSource:0}: Error finding container 23887b3f8e001c99e5f29e9f752e9dc05c45f531139d832a55c9e9c77584eca0: Status 404 returned error can't find the container with id 23887b3f8e001c99e5f29e9f752e9dc05c45f531139d832a55c9e9c77584eca0 Jul 21 21:11:19.734677 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:19.734598 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" event={"ID":"b4b7293a-7545-4cfa-a9e3-74a10253a82d","Type":"ContainerStarted","Data":"23887b3f8e001c99e5f29e9f752e9dc05c45f531139d832a55c9e9c77584eca0"} Jul 21 21:11:20.742803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:20.742758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" event={"ID":"b4b7293a-7545-4cfa-a9e3-74a10253a82d","Type":"ContainerStarted","Data":"2baed62d3624ddd3fd4c8b27d22a0a44bf9f8af4feb0afcbeacc95b0e0e7c51d"} Jul 21 21:11:20.760826 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:20.760772 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" podStartSLOduration=1.549052676 podStartE2EDuration="1.760754841s" totalImagesPullingTime="211.702165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:11:20.760420228 +0000 UTC m=+657.705041316" watchObservedRunningTime="2026-07-21 21:11:20.760754841 +0000 UTC m=+657.705375907" Jul 21 21:11:29.459644 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:29.459605 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:29.460046 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:29.459678 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:29.471787 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:29.471761 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:29.802764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:29.802678 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8" Jul 21 21:11:31.277001 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:31.276937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:11:39.768072 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:11:39.768040 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:12:48.488572 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:12:48.488490 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:12:48.590382 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:12:48.590335 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:12:49.694377 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:12:49.694340 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:12:49.793841 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:12:49.793796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:13:02.235163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:02.235125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:13:06.580556 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:06.580518 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:13:32.404143 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:32.404104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:13:32.502678 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:32.502647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:13:33.477941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:33.477908 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:13:36.611502 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.611425 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-flxpc"] Jul 21 21:13:36.612067 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.611729 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55d6556f9b-flxpc" podUID="a900aa40-d271-4389-9213-8c04fed84e1a" containerName="manager" containerID="cri-o://6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35" gracePeriod=30 Jul 21 21:13:36.849355 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.849332 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:13:36.956306 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.956276 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a900aa40-d271-4389-9213-8c04fed84e1a-webhook-cert\") pod \"a900aa40-d271-4389-9213-8c04fed84e1a\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " Jul 21 21:13:36.956487 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.956409 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc8rx\" (UniqueName: \"kubernetes.io/projected/a900aa40-d271-4389-9213-8c04fed84e1a-kube-api-access-bc8rx\") pod \"a900aa40-d271-4389-9213-8c04fed84e1a\" (UID: \"a900aa40-d271-4389-9213-8c04fed84e1a\") " Jul 21 21:13:36.958396 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.958337 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a900aa40-d271-4389-9213-8c04fed84e1a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a900aa40-d271-4389-9213-8c04fed84e1a" (UID: "a900aa40-d271-4389-9213-8c04fed84e1a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:13:36.958510 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:36.958481 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a900aa40-d271-4389-9213-8c04fed84e1a-kube-api-access-bc8rx" (OuterVolumeSpecName: "kube-api-access-bc8rx") pod "a900aa40-d271-4389-9213-8c04fed84e1a" (UID: "a900aa40-d271-4389-9213-8c04fed84e1a"). InnerVolumeSpecName "kube-api-access-bc8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:13:37.057097 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.057063 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8rx\" (UniqueName: \"kubernetes.io/projected/a900aa40-d271-4389-9213-8c04fed84e1a-kube-api-access-bc8rx\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:13:37.057097 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.057095 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a900aa40-d271-4389-9213-8c04fed84e1a-webhook-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:13:37.475536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.475502 2570 generic.go:356] "Generic (PLEG): container finished" podID="a900aa40-d271-4389-9213-8c04fed84e1a" containerID="6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35" exitCode=0 Jul 21 21:13:37.475709 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.475548 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-flxpc" Jul 21 21:13:37.475709 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.475549 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-flxpc" event={"ID":"a900aa40-d271-4389-9213-8c04fed84e1a","Type":"ContainerDied","Data":"6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35"} Jul 21 21:13:37.475709 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.475682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-flxpc" event={"ID":"a900aa40-d271-4389-9213-8c04fed84e1a","Type":"ContainerDied","Data":"4311ce2fc78c68d98ee425430b3b66c0c2a385ade1667ac6f6e4cf712dc11d43"} Jul 21 21:13:37.475836 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.475719 2570 scope.go:117] "RemoveContainer" containerID="6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35" Jul 21 21:13:37.485752 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.485732 2570 scope.go:117] "RemoveContainer" containerID="6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35" Jul 21 21:13:37.485999 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:13:37.485979 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35\": container with ID starting with 6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35 not found: ID does not exist" containerID="6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35" Jul 21 21:13:37.486039 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.486010 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35"} err="failed to get container status \"6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35\": rpc error: code = NotFound desc = could not find container \"6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35\": container with ID starting with 6b525cdc15c958b7c2cb717157c02c3d14ae3c934ef066a55c2ac2e3e5b96a35 not found: ID does not exist" Jul 21 21:13:37.509615 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.509587 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-flxpc"] Jul 21 21:13:37.516627 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.516607 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-flxpc"] Jul 21 21:13:37.557739 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:13:37.557715 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a900aa40-d271-4389-9213-8c04fed84e1a" path="/var/lib/kubelet/pods/a900aa40-d271-4389-9213-8c04fed84e1a/volumes" Jul 21 21:14:38.408489 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.408455 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55d6556f9b-dnfms"] Jul 21 21:14:38.408960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.408860 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a900aa40-d271-4389-9213-8c04fed84e1a" containerName="manager" Jul 21 21:14:38.408960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.408872 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a900aa40-d271-4389-9213-8c04fed84e1a" containerName="manager" Jul 21 21:14:38.408960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.408943 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a900aa40-d271-4389-9213-8c04fed84e1a" containerName="manager" Jul 21 21:14:38.412062 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.412045 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.414791 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.414766 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 21:14:38.416071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.416053 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qmmzs\"" Jul 21 21:14:38.416177 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.416054 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 21:14:38.427914 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.427850 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-dnfms"] Jul 21 21:14:38.570650 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.570609 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brf4x\" (UniqueName: \"kubernetes.io/projected/b4b95b22-0a3f-4fab-b773-99a834a89d06-kube-api-access-brf4x\") pod \"maas-controller-55d6556f9b-dnfms\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.570650 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.570653 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b95b22-0a3f-4fab-b773-99a834a89d06-webhook-cert\") pod \"maas-controller-55d6556f9b-dnfms\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.671940 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.671909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brf4x\" (UniqueName: \"kubernetes.io/projected/b4b95b22-0a3f-4fab-b773-99a834a89d06-kube-api-access-brf4x\") pod \"maas-controller-55d6556f9b-dnfms\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.672087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.671948 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b95b22-0a3f-4fab-b773-99a834a89d06-webhook-cert\") pod \"maas-controller-55d6556f9b-dnfms\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.674512 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.674489 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b95b22-0a3f-4fab-b773-99a834a89d06-webhook-cert\") pod \"maas-controller-55d6556f9b-dnfms\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.680655 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.680628 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brf4x\" (UniqueName: \"kubernetes.io/projected/b4b95b22-0a3f-4fab-b773-99a834a89d06-kube-api-access-brf4x\") pod \"maas-controller-55d6556f9b-dnfms\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.723907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.723885 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:38.859785 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:38.859750 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-dnfms"] Jul 21 21:14:38.861196 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:14:38.861168 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b95b22_0a3f_4fab_b773_99a834a89d06.slice/crio-b1713398de34a3d8f6c7aada9eb562cbf8a93a72b70035a8eca8022729e0d206 WatchSource:0}: Error finding container b1713398de34a3d8f6c7aada9eb562cbf8a93a72b70035a8eca8022729e0d206: Status 404 returned error can't find the container with id b1713398de34a3d8f6c7aada9eb562cbf8a93a72b70035a8eca8022729e0d206 Jul 21 21:14:39.803996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:39.803958 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-dnfms" event={"ID":"b4b95b22-0a3f-4fab-b773-99a834a89d06","Type":"ContainerStarted","Data":"07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c"} Jul 21 21:14:39.804431 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:39.804012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-dnfms" event={"ID":"b4b95b22-0a3f-4fab-b773-99a834a89d06","Type":"ContainerStarted","Data":"b1713398de34a3d8f6c7aada9eb562cbf8a93a72b70035a8eca8022729e0d206"} Jul 21 21:14:39.804431 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:39.804109 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:39.823014 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:39.822966 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55d6556f9b-dnfms" podStartSLOduration=1.342577452 podStartE2EDuration="1.822950384s" totalImagesPullingTime="480.372932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:14:39.820752475 +0000 UTC m=+856.765373541" watchObservedRunningTime="2026-07-21 21:14:39.822950384 +0000 UTC m=+856.767571476" Jul 21 21:14:50.812006 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:50.811973 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:55.387958 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.387924 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:14:55.509420 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.509390 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-dnfms"] Jul 21 21:14:55.509615 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.509589 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55d6556f9b-dnfms" podUID="b4b95b22-0a3f-4fab-b773-99a834a89d06" containerName="manager" containerID="cri-o://07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c" gracePeriod=30 Jul 21 21:14:55.764247 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.764223 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:55.828188 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.828159 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b95b22-0a3f-4fab-b773-99a834a89d06-webhook-cert\") pod \"b4b95b22-0a3f-4fab-b773-99a834a89d06\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " Jul 21 21:14:55.830408 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.830380 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b95b22-0a3f-4fab-b773-99a834a89d06-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b4b95b22-0a3f-4fab-b773-99a834a89d06" (UID: "b4b95b22-0a3f-4fab-b773-99a834a89d06"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:14:55.893680 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.893646 2570 generic.go:356] "Generic (PLEG): container finished" podID="b4b95b22-0a3f-4fab-b773-99a834a89d06" containerID="07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c" exitCode=0 Jul 21 21:14:55.893827 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.893686 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-dnfms" Jul 21 21:14:55.893827 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.893722 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-dnfms" event={"ID":"b4b95b22-0a3f-4fab-b773-99a834a89d06","Type":"ContainerDied","Data":"07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c"} Jul 21 21:14:55.893827 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.893775 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-dnfms" event={"ID":"b4b95b22-0a3f-4fab-b773-99a834a89d06","Type":"ContainerDied","Data":"b1713398de34a3d8f6c7aada9eb562cbf8a93a72b70035a8eca8022729e0d206"} Jul 21 21:14:55.893827 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.893804 2570 scope.go:117] "RemoveContainer" containerID="07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c" Jul 21 21:14:55.902668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.902648 2570 scope.go:117] "RemoveContainer" containerID="07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c" Jul 21 21:14:55.902913 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:14:55.902894 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c\": container with ID starting with 07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c not found: ID does not exist" containerID="07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c" Jul 21 21:14:55.902960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.902933 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c"} err="failed to get container status \"07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c\": rpc error: code = NotFound desc = could not find container \"07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c\": container with ID starting with 07adab5085a7c463ae718afa6e7c2bdd1d554b6d1b3fde10c07bb3af6d0c115c not found: ID does not exist" Jul 21 21:14:55.929458 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.929438 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brf4x\" (UniqueName: \"kubernetes.io/projected/b4b95b22-0a3f-4fab-b773-99a834a89d06-kube-api-access-brf4x\") pod \"b4b95b22-0a3f-4fab-b773-99a834a89d06\" (UID: \"b4b95b22-0a3f-4fab-b773-99a834a89d06\") " Jul 21 21:14:55.929622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.929610 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4b95b22-0a3f-4fab-b773-99a834a89d06-webhook-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:14:55.931471 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:55.931416 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b95b22-0a3f-4fab-b773-99a834a89d06-kube-api-access-brf4x" (OuterVolumeSpecName: "kube-api-access-brf4x") pod "b4b95b22-0a3f-4fab-b773-99a834a89d06" (UID: "b4b95b22-0a3f-4fab-b773-99a834a89d06"). InnerVolumeSpecName "kube-api-access-brf4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:14:56.030141 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:56.030105 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brf4x\" (UniqueName: \"kubernetes.io/projected/b4b95b22-0a3f-4fab-b773-99a834a89d06-kube-api-access-brf4x\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:14:56.218420 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:56.218391 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-dnfms"] Jul 21 21:14:56.222803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:56.222778 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-dnfms"] Jul 21 21:14:56.928484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:56.928428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:14:57.062216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.062182 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55d6556f9b-bf9cf"] Jul 21 21:14:57.062637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.062619 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b95b22-0a3f-4fab-b773-99a834a89d06" containerName="manager" Jul 21 21:14:57.062699 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.062640 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b95b22-0a3f-4fab-b773-99a834a89d06" containerName="manager" Jul 21 21:14:57.062804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.062787 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b95b22-0a3f-4fab-b773-99a834a89d06" containerName="manager" Jul 21 21:14:57.067438 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.067420 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.070529 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.070509 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 21:14:57.070641 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.070553 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 21:14:57.071671 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.071653 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qmmzs\"" Jul 21 21:14:57.078274 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.078251 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-bf9cf"] Jul 21 21:14:57.141707 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.141675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5258ce45-e3fe-4709-a616-63a72bd46073-webhook-cert\") pod \"maas-controller-55d6556f9b-bf9cf\" (UID: \"5258ce45-e3fe-4709-a616-63a72bd46073\") " pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.141899 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.141724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj7x\" (UniqueName: \"kubernetes.io/projected/5258ce45-e3fe-4709-a616-63a72bd46073-kube-api-access-ghj7x\") pod \"maas-controller-55d6556f9b-bf9cf\" (UID: \"5258ce45-e3fe-4709-a616-63a72bd46073\") " pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.243218 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.243121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5258ce45-e3fe-4709-a616-63a72bd46073-webhook-cert\") pod \"maas-controller-55d6556f9b-bf9cf\" (UID: \"5258ce45-e3fe-4709-a616-63a72bd46073\") " pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.243218 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.243172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj7x\" (UniqueName: \"kubernetes.io/projected/5258ce45-e3fe-4709-a616-63a72bd46073-kube-api-access-ghj7x\") pod \"maas-controller-55d6556f9b-bf9cf\" (UID: \"5258ce45-e3fe-4709-a616-63a72bd46073\") " pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.245772 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.245751 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5258ce45-e3fe-4709-a616-63a72bd46073-webhook-cert\") pod \"maas-controller-55d6556f9b-bf9cf\" (UID: \"5258ce45-e3fe-4709-a616-63a72bd46073\") " pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.253212 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.253190 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj7x\" (UniqueName: \"kubernetes.io/projected/5258ce45-e3fe-4709-a616-63a72bd46073-kube-api-access-ghj7x\") pod \"maas-controller-55d6556f9b-bf9cf\" (UID: \"5258ce45-e3fe-4709-a616-63a72bd46073\") " pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.380504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.380469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:57.518021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.517997 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55d6556f9b-bf9cf"] Jul 21 21:14:57.519669 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:14:57.519639 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5258ce45_e3fe_4709_a616_63a72bd46073.slice/crio-096fd5a99d66687c4292c996936484ed5e0f28b62c221341c9efa733ad8c0603 WatchSource:0}: Error finding container 096fd5a99d66687c4292c996936484ed5e0f28b62c221341c9efa733ad8c0603: Status 404 returned error can't find the container with id 096fd5a99d66687c4292c996936484ed5e0f28b62c221341c9efa733ad8c0603 Jul 21 21:14:57.557064 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.557028 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b95b22-0a3f-4fab-b773-99a834a89d06" path="/var/lib/kubelet/pods/b4b95b22-0a3f-4fab-b773-99a834a89d06/volumes" Jul 21 21:14:57.910222 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:57.910195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" event={"ID":"5258ce45-e3fe-4709-a616-63a72bd46073","Type":"ContainerStarted","Data":"096fd5a99d66687c4292c996936484ed5e0f28b62c221341c9efa733ad8c0603"} Jul 21 21:14:58.917339 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:58.917301 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" event={"ID":"5258ce45-e3fe-4709-a616-63a72bd46073","Type":"ContainerStarted","Data":"5aab069e6fb0c4408e2bbb23619eb069eacb9d73f0cda26579edd2d42b4bd81e"} Jul 21 21:14:58.917736 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:58.917386 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:14:58.935430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:14:58.935350 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" podStartSLOduration=1.568238838 podStartE2EDuration="1.935336139s" totalImagesPullingTime="367.097301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:14:58.934148775 +0000 UTC m=+875.878769841" watchObservedRunningTime="2026-07-21 21:14:58.935336139 +0000 UTC m=+875.879957204" Jul 21 21:15:09.924787 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:09.924759 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55d6556f9b-bf9cf" Jul 21 21:15:33.445179 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:33.445142 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:33.544772 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:33.544737 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:33.688663 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:33.688627 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:39.749630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:39.749598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:39.856892 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:39.856851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:39.949322 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:39.949294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:40.079076 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:40.078995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:46.210065 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:46.210010 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:15:46.269934 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:15:46.269904 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:16:44.504278 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:16:44.504241 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:19:07.796991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:19:07.796947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:19:57.726410 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:19:57.726370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:19:57.828006 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:19:57.827947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:22:27.466179 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:22:27.466144 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:23:21.002029 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:23:21.001972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:23:32.757290 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:23:32.757236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:23:53.446840 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:23:53.446790 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:23:54.196445 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:23:54.196405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:25:20.156856 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:25:20.156824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:26:50.854475 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:26:50.854440 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:04.956711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:04.956677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:28.584083 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:28.583998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:31.722335 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:31.722300 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:32.885259 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:32.885218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:38.054186 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:38.054152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:38.152983 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:38.152949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:38.298892 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:38.298858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:38.622163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:38.622132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:39.327681 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:39.327653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:40.608559 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.608523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:40.687057 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.687026 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc"] Jul 21 21:27:40.690603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.690585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.692950 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.692928 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:40.693344 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.693324 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-82b9182c-serving-cert\"" Jul 21 21:27:40.699580 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.699557 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc"] Jul 21 21:27:40.732205 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.732164 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.732385 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.732292 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-openshift-service-ca\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.732385 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.732324 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz58m\" (UniqueName: \"kubernetes.io/projected/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-kube-api-access-dz58m\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.833336 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.833300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.833536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.833387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-openshift-service-ca\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.833536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.833407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz58m\" (UniqueName: \"kubernetes.io/projected/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-kube-api-access-dz58m\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.833536 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:27:40.833487 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-serving-cert: secret "maas-api-e2e-ait-82b9182c-serving-cert" not found Jul 21 21:27:40.833708 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:27:40.833590 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls podName:7fbeaa5b-e044-4a11-81cc-0de2c3c19c17 nodeName:}" failed. No retries permitted until 2026-07-21 21:27:41.333568506 +0000 UTC m=+1638.278189567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls") pod "maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" (UID: "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17") : secret "maas-api-e2e-ait-82b9182c-serving-cert" not found Jul 21 21:27:40.834117 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.834098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-openshift-service-ca\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:40.842333 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:40.842312 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz58m\" (UniqueName: \"kubernetes.io/projected/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-kube-api-access-dz58m\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:41.092270 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.092233 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:41.230956 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.230921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:41.338682 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.338644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:41.341205 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.341188 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls\") pod \"maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:41.430208 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.430173 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:41.530427 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.530354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:27:41.602579 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.602544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:41.747808 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.747783 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc"] Jul 21 21:27:41.750249 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:27:41.750209 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbeaa5b_e044_4a11_81cc_0de2c3c19c17.slice/crio-e5676cf10a277c8ba15341b5b4d2176075b94aa733aa96eaeffbcfd82868bbf7 WatchSource:0}: Error finding container e5676cf10a277c8ba15341b5b4d2176075b94aa733aa96eaeffbcfd82868bbf7: Status 404 returned error can't find the container with id e5676cf10a277c8ba15341b5b4d2176075b94aa733aa96eaeffbcfd82868bbf7 Jul 21 21:27:41.751714 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:41.751696 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:27:42.022013 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:42.021919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" event={"ID":"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17","Type":"ContainerStarted","Data":"e5676cf10a277c8ba15341b5b4d2176075b94aa733aa96eaeffbcfd82868bbf7"} Jul 21 21:27:43.028847 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:43.028816 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" event={"ID":"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17","Type":"ContainerStarted","Data":"0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e"} Jul 21 21:27:43.029245 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:43.028941 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:27:43.049667 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:43.049616 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" podStartSLOduration=2.263069627 podStartE2EDuration="3.049600916s" totalImagesPullingTime="786.531289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:27:43.046693991 +0000 UTC m=+1639.991315055" watchObservedRunningTime="2026-07-21 21:27:43.049600916 +0000 UTC m=+1639.994221980" Jul 21 21:27:49.041419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:27:49.041387 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:28:16.655547 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:16.655511 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc"] Jul 21 21:28:16.656454 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:16.656260 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" podUID="7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" containerName="maas-api" containerID="cri-o://0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e" gracePeriod=30 Jul 21 21:28:16.689996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:16.689968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:16.910769 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:16.910714 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:28:17.063788 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.063753 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-openshift-service-ca\") pod \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " Jul 21 21:28:17.063974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.063916 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz58m\" (UniqueName: \"kubernetes.io/projected/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-kube-api-access-dz58m\") pod \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " Jul 21 21:28:17.063974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.063946 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls\") pod \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\" (UID: \"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17\") " Jul 21 21:28:17.064161 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.064135 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" (UID: "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:28:17.064279 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.064263 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:28:17.066289 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.066262 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" (UID: "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:28:17.066411 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.066324 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-kube-api-access-dz58m" (OuterVolumeSpecName: "kube-api-access-dz58m") pod "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" (UID: "7fbeaa5b-e044-4a11-81cc-0de2c3c19c17"). InnerVolumeSpecName "kube-api-access-dz58m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:28:17.165008 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.164978 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz58m\" (UniqueName: \"kubernetes.io/projected/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-kube-api-access-dz58m\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:28:17.165008 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.165009 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:28:17.208222 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.208189 2570 generic.go:356] "Generic (PLEG): container finished" podID="7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" containerID="0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e" exitCode=0 Jul 21 21:28:17.208404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.208238 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" Jul 21 21:28:17.208404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.208281 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" event={"ID":"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17","Type":"ContainerDied","Data":"0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e"} Jul 21 21:28:17.208404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.208335 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc" event={"ID":"7fbeaa5b-e044-4a11-81cc-0de2c3c19c17","Type":"ContainerDied","Data":"e5676cf10a277c8ba15341b5b4d2176075b94aa733aa96eaeffbcfd82868bbf7"} Jul 21 21:28:17.208404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.208385 2570 scope.go:117] "RemoveContainer" containerID="0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e" Jul 21 21:28:17.219075 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.218999 2570 scope.go:117] "RemoveContainer" containerID="0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e" Jul 21 21:28:17.221024 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:28:17.221001 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e\": container with ID starting with 0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e not found: ID does not exist" containerID="0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e" Jul 21 21:28:17.221147 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.221033 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e"} err="failed to get container status \"0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e\": rpc error: code = NotFound desc = could not find container \"0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e\": container with ID starting with 0716357f28824507cdfd2e4a5454c297a4123869ab49a809d295bf106b67e51e not found: ID does not exist" Jul 21 21:28:17.234477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.234379 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc"] Jul 21 21:28:17.237302 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.237282 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82b9182c-85f79ffbf9-n66lc"] Jul 21 21:28:17.557590 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:17.557493 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" path="/var/lib/kubelet/pods/7fbeaa5b-e044-4a11-81cc-0de2c3c19c17/volumes" Jul 21 21:28:21.338422 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:21.338381 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:21.515527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:21.515483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:21.768144 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:21.768113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:22.028578 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:22.028498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:22.127449 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:22.127418 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:22.255906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:22.255863 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:22.604207 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:22.604171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:22.919049 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:22.919013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:24.875814 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:24.875755 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:25.048923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.048890 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b"] Jul 21 21:28:25.049370 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.049344 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" containerName="maas-api" Jul 21 21:28:25.049533 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.049380 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" containerName="maas-api" Jul 21 21:28:25.049533 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.049526 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbeaa5b-e044-4a11-81cc-0de2c3c19c17" containerName="maas-api" Jul 21 21:28:25.052487 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.052467 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.055011 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.054991 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1710f404-serving-cert\"" Jul 21 21:28:25.064216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.064192 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b"] Jul 21 21:28:25.080028 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.079992 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:25.136277 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.136185 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.136277 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.136253 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvv6\" (UniqueName: \"kubernetes.io/projected/28e18724-6ad1-4799-a755-b8384f92ea1a-kube-api-access-frvv6\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.136511 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.136291 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e18724-6ad1-4799-a755-b8384f92ea1a-openshift-service-ca\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.237240 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.237202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.237469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.237262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvv6\" (UniqueName: \"kubernetes.io/projected/28e18724-6ad1-4799-a755-b8384f92ea1a-kube-api-access-frvv6\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.237469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.237286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e18724-6ad1-4799-a755-b8384f92ea1a-openshift-service-ca\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.237469 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:28:25.237398 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-serving-cert: secret "maas-api-e2e-migrate-1710f404-serving-cert" not found Jul 21 21:28:25.237668 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:28:25.237495 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls podName:28e18724-6ad1-4799-a755-b8384f92ea1a nodeName:}" failed. No retries permitted until 2026-07-21 21:28:25.737472587 +0000 UTC m=+1682.682093639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls") pod "maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" (UID: "28e18724-6ad1-4799-a755-b8384f92ea1a") : secret "maas-api-e2e-migrate-1710f404-serving-cert" not found Jul 21 21:28:25.237939 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.237919 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e18724-6ad1-4799-a755-b8384f92ea1a-openshift-service-ca\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.249768 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.249743 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvv6\" (UniqueName: \"kubernetes.io/projected/28e18724-6ad1-4799-a755-b8384f92ea1a-kube-api-access-frvv6\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.289741 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.289709 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:25.555708 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.555671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:25.656924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.656896 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:28:25.741669 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.741640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.744071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.744044 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls\") pod \"maas-api-e2e-migrate-1710f404-78d749477d-s2l6b\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:25.964686 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:25.964642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:26.314456 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:26.314421 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b"] Jul 21 21:28:26.315568 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:28:26.315542 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e18724_6ad1_4799_a755_b8384f92ea1a.slice/crio-3457856e9cb22337907f11e898a1c4bac4b4a872b2b7e2f81620839baf3c917f WatchSource:0}: Error finding container 3457856e9cb22337907f11e898a1c4bac4b4a872b2b7e2f81620839baf3c917f: Status 404 returned error can't find the container with id 3457856e9cb22337907f11e898a1c4bac4b4a872b2b7e2f81620839baf3c917f Jul 21 21:28:27.272998 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:27.272964 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" event={"ID":"28e18724-6ad1-4799-a755-b8384f92ea1a","Type":"ContainerStarted","Data":"972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c"} Jul 21 21:28:27.273425 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:27.273027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" event={"ID":"28e18724-6ad1-4799-a755-b8384f92ea1a","Type":"ContainerStarted","Data":"3457856e9cb22337907f11e898a1c4bac4b4a872b2b7e2f81620839baf3c917f"} Jul 21 21:28:27.273425 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:27.273054 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:27.292564 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:27.292516 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" podStartSLOduration=1.807532994 podStartE2EDuration="2.292500692s" totalImagesPullingTime="484.967698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:28:27.291533346 +0000 UTC m=+1684.236154437" watchObservedRunningTime="2026-07-21 21:28:27.292500692 +0000 UTC m=+1684.237121756" Jul 21 21:28:33.283993 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:33.283962 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:28:41.007220 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:28:41.007168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:00.408061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.408023 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b"] Jul 21 21:29:00.408635 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.408277 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" podUID="28e18724-6ad1-4799-a755-b8384f92ea1a" containerName="maas-api" containerID="cri-o://972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c" gracePeriod=30 Jul 21 21:29:00.455535 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.455499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:00.654634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.654606 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:29:00.743302 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.743257 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls\") pod \"28e18724-6ad1-4799-a755-b8384f92ea1a\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " Jul 21 21:29:00.743530 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.743410 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e18724-6ad1-4799-a755-b8384f92ea1a-openshift-service-ca\") pod \"28e18724-6ad1-4799-a755-b8384f92ea1a\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " Jul 21 21:29:00.743530 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.743503 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frvv6\" (UniqueName: \"kubernetes.io/projected/28e18724-6ad1-4799-a755-b8384f92ea1a-kube-api-access-frvv6\") pod \"28e18724-6ad1-4799-a755-b8384f92ea1a\" (UID: \"28e18724-6ad1-4799-a755-b8384f92ea1a\") " Jul 21 21:29:00.743846 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.743820 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e18724-6ad1-4799-a755-b8384f92ea1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28e18724-6ad1-4799-a755-b8384f92ea1a" (UID: "28e18724-6ad1-4799-a755-b8384f92ea1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:29:00.745603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.745577 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28e18724-6ad1-4799-a755-b8384f92ea1a" (UID: "28e18724-6ad1-4799-a755-b8384f92ea1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:29:00.745713 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.745692 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e18724-6ad1-4799-a755-b8384f92ea1a-kube-api-access-frvv6" (OuterVolumeSpecName: "kube-api-access-frvv6") pod "28e18724-6ad1-4799-a755-b8384f92ea1a" (UID: "28e18724-6ad1-4799-a755-b8384f92ea1a"). InnerVolumeSpecName "kube-api-access-frvv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:29:00.844485 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.844441 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvv6\" (UniqueName: \"kubernetes.io/projected/28e18724-6ad1-4799-a755-b8384f92ea1a-kube-api-access-frvv6\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:29:00.844485 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.844478 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28e18724-6ad1-4799-a755-b8384f92ea1a-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:29:00.844485 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:00.844488 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e18724-6ad1-4799-a755-b8384f92ea1a-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:29:01.459773 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.459738 2570 generic.go:356] "Generic (PLEG): container finished" podID="28e18724-6ad1-4799-a755-b8384f92ea1a" containerID="972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c" exitCode=0 Jul 21 21:29:01.460263 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.459788 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" Jul 21 21:29:01.460263 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.459824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" event={"ID":"28e18724-6ad1-4799-a755-b8384f92ea1a","Type":"ContainerDied","Data":"972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c"} Jul 21 21:29:01.460263 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.459885 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b" event={"ID":"28e18724-6ad1-4799-a755-b8384f92ea1a","Type":"ContainerDied","Data":"3457856e9cb22337907f11e898a1c4bac4b4a872b2b7e2f81620839baf3c917f"} Jul 21 21:29:01.460263 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.459916 2570 scope.go:117] "RemoveContainer" containerID="972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c" Jul 21 21:29:01.470763 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.470739 2570 scope.go:117] "RemoveContainer" containerID="972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c" Jul 21 21:29:01.471076 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:29:01.471052 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c\": container with ID starting with 972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c not found: ID does not exist" containerID="972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c" Jul 21 21:29:01.471153 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.471090 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c"} err="failed to get container status \"972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c\": rpc error: code = NotFound desc = could not find container \"972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c\": container with ID starting with 972b69d39ad4891c2098050ed02d9d50f412871559ff6984ea1ba35bd039c41c not found: ID does not exist" Jul 21 21:29:01.482145 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.482108 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b"] Jul 21 21:29:01.484573 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.484550 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1710f404-78d749477d-s2l6b"] Jul 21 21:29:01.557448 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:01.557405 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e18724-6ad1-4799-a755-b8384f92ea1a" path="/var/lib/kubelet/pods/28e18724-6ad1-4799-a755-b8384f92ea1a/volumes" Jul 21 21:29:05.017630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:05.017593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:05.195189 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:05.195160 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:05.426466 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:05.426420 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:05.666559 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:05.666527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:06.007300 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:06.007266 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:06.190166 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:06.190131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:07.965407 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:07.965348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:08.071496 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.071463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:08.072889 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.072863 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6"] Jul 21 21:29:08.073574 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.073554 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e18724-6ad1-4799-a755-b8384f92ea1a" containerName="maas-api" Jul 21 21:29:08.073654 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.073579 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e18724-6ad1-4799-a755-b8384f92ea1a" containerName="maas-api" Jul 21 21:29:08.073713 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.073688 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e18724-6ad1-4799-a755-b8384f92ea1a" containerName="maas-api" Jul 21 21:29:08.078166 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.078143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.081176 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.081155 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4a82f5f-serving-cert\"" Jul 21 21:29:08.087939 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.087914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6"] Jul 21 21:29:08.150616 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.150582 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:08.211179 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.211144 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzcz\" (UniqueName: \"kubernetes.io/projected/5d9c7480-5ef0-4e2a-93ba-88cea214d694-kube-api-access-rmzcz\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.211339 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.211193 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9c7480-5ef0-4e2a-93ba-88cea214d694-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.211339 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.211222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.312244 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.312163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzcz\" (UniqueName: \"kubernetes.io/projected/5d9c7480-5ef0-4e2a-93ba-88cea214d694-kube-api-access-rmzcz\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.312244 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.312215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9c7480-5ef0-4e2a-93ba-88cea214d694-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.312453 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.312243 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.312453 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:29:08.312353 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-serving-cert: secret "maas-api-e2e-ait-a4a82f5f-serving-cert" not found Jul 21 21:29:08.312453 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:29:08.312442 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls podName:5d9c7480-5ef0-4e2a-93ba-88cea214d694 nodeName:}" failed. No retries permitted until 2026-07-21 21:29:08.81242311 +0000 UTC m=+1725.757044158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls") pod "maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" (UID: "5d9c7480-5ef0-4e2a-93ba-88cea214d694") : secret "maas-api-e2e-ait-a4a82f5f-serving-cert" not found Jul 21 21:29:08.312986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.312966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9c7480-5ef0-4e2a-93ba-88cea214d694-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.323162 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.323134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzcz\" (UniqueName: \"kubernetes.io/projected/5d9c7480-5ef0-4e2a-93ba-88cea214d694-kube-api-access-rmzcz\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.741422 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.741386 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:08.818200 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.818170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.820811 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.820788 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls\") pod \"maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:08.835289 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.835263 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:08.989527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:08.989486 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:09.134738 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:09.134712 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6"] Jul 21 21:29:09.136421 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:29:09.136395 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9c7480_5ef0_4e2a_93ba_88cea214d694.slice/crio-9a09a34f5d0b543df1e146f9b2ec6a7721899fa05830678594e4c6ca246462c1 WatchSource:0}: Error finding container 9a09a34f5d0b543df1e146f9b2ec6a7721899fa05830678594e4c6ca246462c1: Status 404 returned error can't find the container with id 9a09a34f5d0b543df1e146f9b2ec6a7721899fa05830678594e4c6ca246462c1 Jul 21 21:29:09.506826 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:09.506788 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" event={"ID":"5d9c7480-5ef0-4e2a-93ba-88cea214d694","Type":"ContainerStarted","Data":"9a09a34f5d0b543df1e146f9b2ec6a7721899fa05830678594e4c6ca246462c1"} Jul 21 21:29:10.513867 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:10.513824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" event={"ID":"5d9c7480-5ef0-4e2a-93ba-88cea214d694","Type":"ContainerStarted","Data":"0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024"} Jul 21 21:29:10.514322 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:10.513969 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:10.532589 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:10.532534 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" podStartSLOduration=2.104773101 podStartE2EDuration="2.532520218s" totalImagesPullingTime="427.747117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:29:10.530600534 +0000 UTC m=+1727.475221599" watchObservedRunningTime="2026-07-21 21:29:10.532520218 +0000 UTC m=+1727.477141283" Jul 21 21:29:16.524969 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:16.524940 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:44.276303 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.276262 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6"] Jul 21 21:29:44.276848 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.276640 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" podUID="5d9c7480-5ef0-4e2a-93ba-88cea214d694" containerName="maas-api" containerID="cri-o://0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024" gracePeriod=30 Jul 21 21:29:44.300598 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.300569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:44.525982 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.525958 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:44.646688 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.646606 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9c7480-5ef0-4e2a-93ba-88cea214d694-openshift-service-ca\") pod \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " Jul 21 21:29:44.646837 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.646710 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls\") pod \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " Jul 21 21:29:44.646837 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.646731 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmzcz\" (UniqueName: \"kubernetes.io/projected/5d9c7480-5ef0-4e2a-93ba-88cea214d694-kube-api-access-rmzcz\") pod \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\" (UID: \"5d9c7480-5ef0-4e2a-93ba-88cea214d694\") " Jul 21 21:29:44.647018 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.646994 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9c7480-5ef0-4e2a-93ba-88cea214d694-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d9c7480-5ef0-4e2a-93ba-88cea214d694" (UID: "5d9c7480-5ef0-4e2a-93ba-88cea214d694"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:29:44.649007 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.648976 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9c7480-5ef0-4e2a-93ba-88cea214d694-kube-api-access-rmzcz" (OuterVolumeSpecName: "kube-api-access-rmzcz") pod "5d9c7480-5ef0-4e2a-93ba-88cea214d694" (UID: "5d9c7480-5ef0-4e2a-93ba-88cea214d694"). InnerVolumeSpecName "kube-api-access-rmzcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:29:44.649126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.649105 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d9c7480-5ef0-4e2a-93ba-88cea214d694" (UID: "5d9c7480-5ef0-4e2a-93ba-88cea214d694"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:29:44.703625 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.703590 2570 generic.go:356] "Generic (PLEG): container finished" podID="5d9c7480-5ef0-4e2a-93ba-88cea214d694" containerID="0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024" exitCode=0 Jul 21 21:29:44.703803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.703644 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" Jul 21 21:29:44.703803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.703685 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" event={"ID":"5d9c7480-5ef0-4e2a-93ba-88cea214d694","Type":"ContainerDied","Data":"0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024"} Jul 21 21:29:44.703803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.703743 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6" event={"ID":"5d9c7480-5ef0-4e2a-93ba-88cea214d694","Type":"ContainerDied","Data":"9a09a34f5d0b543df1e146f9b2ec6a7721899fa05830678594e4c6ca246462c1"} Jul 21 21:29:44.703803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.703770 2570 scope.go:117] "RemoveContainer" containerID="0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024" Jul 21 21:29:44.713273 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.713257 2570 scope.go:117] "RemoveContainer" containerID="0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024" Jul 21 21:29:44.713563 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:29:44.713537 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024\": container with ID starting with 0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024 not found: ID does not exist" containerID="0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024" Jul 21 21:29:44.713663 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.713571 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024"} err="failed to get container status \"0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024\": rpc error: code = NotFound desc = could not find container \"0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024\": container with ID starting with 0de34560084aa91431887d467495a2bc878947bf34771c73187565564960d024 not found: ID does not exist" Jul 21 21:29:44.726135 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.726113 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6"] Jul 21 21:29:44.732469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.732449 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a82f5f-786fbbf6c7-726x6"] Jul 21 21:29:44.747941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.747918 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9c7480-5ef0-4e2a-93ba-88cea214d694-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:29:44.747941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.747939 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzcz\" (UniqueName: \"kubernetes.io/projected/5d9c7480-5ef0-4e2a-93ba-88cea214d694-kube-api-access-rmzcz\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:29:44.748055 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:44.747948 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9c7480-5ef0-4e2a-93ba-88cea214d694-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:29:45.556430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:45.556397 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9c7480-5ef0-4e2a-93ba-88cea214d694" path="/var/lib/kubelet/pods/5d9c7480-5ef0-4e2a-93ba-88cea214d694/volumes" Jul 21 21:29:48.953320 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:48.953286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:50.266640 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:50.266607 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:50.530875 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:50.530797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:50.776136 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:50.776103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:51.128473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:51.127698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:51.412637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:51.412549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:53.217990 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.217941 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:53.331861 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.331822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:53.361415 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.361385 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp"] Jul 21 21:29:53.361784 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.361770 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9c7480-5ef0-4e2a-93ba-88cea214d694" containerName="maas-api" Jul 21 21:29:53.361784 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.361784 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9c7480-5ef0-4e2a-93ba-88cea214d694" containerName="maas-api" Jul 21 21:29:53.361909 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.361882 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9c7480-5ef0-4e2a-93ba-88cea214d694" containerName="maas-api" Jul 21 21:29:53.366267 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.366248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.370381 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.370344 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e49e6eb7-serving-cert\"" Jul 21 21:29:53.379567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.379541 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp"] Jul 21 21:29:53.424722 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.424680 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:53.527069 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.526981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.527069 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.527060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47188b56-9ee4-4fab-99cb-6a205b2facb1-openshift-service-ca\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.527252 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.527112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpff\" (UniqueName: \"kubernetes.io/projected/47188b56-9ee4-4fab-99cb-6a205b2facb1-kube-api-access-vvpff\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.628457 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.628420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47188b56-9ee4-4fab-99cb-6a205b2facb1-openshift-service-ca\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.628457 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.628460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpff\" (UniqueName: \"kubernetes.io/projected/47188b56-9ee4-4fab-99cb-6a205b2facb1-kube-api-access-vvpff\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.628691 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.628501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.628691 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:29:53.628626 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-serving-cert: secret "maas-api-e2e-derive-e49e6eb7-serving-cert" not found Jul 21 21:29:53.628691 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:29:53.628682 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls podName:47188b56-9ee4-4fab-99cb-6a205b2facb1 nodeName:}" failed. No retries permitted until 2026-07-21 21:29:54.128665325 +0000 UTC m=+1771.073286368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls") pod "maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" (UID: "47188b56-9ee4-4fab-99cb-6a205b2facb1") : secret "maas-api-e2e-derive-e49e6eb7-serving-cert" not found Jul 21 21:29:53.629087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.629067 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47188b56-9ee4-4fab-99cb-6a205b2facb1-openshift-service-ca\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.637824 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.637798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpff\" (UniqueName: \"kubernetes.io/projected/47188b56-9ee4-4fab-99cb-6a205b2facb1-kube-api-access-vvpff\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:53.884919 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:53.884815 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:54.007423 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.007389 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:54.110276 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.110240 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:29:54.132622 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.132595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:54.135221 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.135149 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls\") pod \"maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:54.278277 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.278238 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:54.427835 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.427808 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp"] Jul 21 21:29:54.429515 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:29:54.429483 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47188b56_9ee4_4fab_99cb_6a205b2facb1.slice/crio-f382c78b71582722410f12e27eda45bf82f1e552bfa87abae8eddf96fbd0448b WatchSource:0}: Error finding container f382c78b71582722410f12e27eda45bf82f1e552bfa87abae8eddf96fbd0448b: Status 404 returned error can't find the container with id f382c78b71582722410f12e27eda45bf82f1e552bfa87abae8eddf96fbd0448b Jul 21 21:29:54.766122 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:54.766038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" event={"ID":"47188b56-9ee4-4fab-99cb-6a205b2facb1","Type":"ContainerStarted","Data":"f382c78b71582722410f12e27eda45bf82f1e552bfa87abae8eddf96fbd0448b"} Jul 21 21:29:55.772930 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:55.772896 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" event={"ID":"47188b56-9ee4-4fab-99cb-6a205b2facb1","Type":"ContainerStarted","Data":"c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749"} Jul 21 21:29:55.773443 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:55.772956 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:29:55.794842 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:29:55.794792 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" podStartSLOduration=2.446555295 podStartE2EDuration="2.794778935s" totalImagesPullingTime="348.22364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:29:55.792613059 +0000 UTC m=+1772.737234126" watchObservedRunningTime="2026-07-21 21:29:55.794778935 +0000 UTC m=+1772.739399999" Jul 21 21:30:01.784218 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:01.784187 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:30:20.886326 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:20.886246 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:21.005210 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:21.005175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:28.878541 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:28.878504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:28.977123 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:28.977070 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:29.095122 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.095063 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp"] Jul 21 21:30:29.095389 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.095319 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" podUID="47188b56-9ee4-4fab-99cb-6a205b2facb1" containerName="maas-api" containerID="cri-o://c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749" gracePeriod=30 Jul 21 21:30:29.126630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.126601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:29.351043 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.351016 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:30:29.435907 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.435877 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls\") pod \"47188b56-9ee4-4fab-99cb-6a205b2facb1\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " Jul 21 21:30:29.436087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.435925 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47188b56-9ee4-4fab-99cb-6a205b2facb1-openshift-service-ca\") pod \"47188b56-9ee4-4fab-99cb-6a205b2facb1\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " Jul 21 21:30:29.436087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.436012 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvpff\" (UniqueName: \"kubernetes.io/projected/47188b56-9ee4-4fab-99cb-6a205b2facb1-kube-api-access-vvpff\") pod \"47188b56-9ee4-4fab-99cb-6a205b2facb1\" (UID: \"47188b56-9ee4-4fab-99cb-6a205b2facb1\") " Jul 21 21:30:29.436399 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.436348 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47188b56-9ee4-4fab-99cb-6a205b2facb1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47188b56-9ee4-4fab-99cb-6a205b2facb1" (UID: "47188b56-9ee4-4fab-99cb-6a205b2facb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:30:29.438209 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.438184 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47188b56-9ee4-4fab-99cb-6a205b2facb1-kube-api-access-vvpff" (OuterVolumeSpecName: "kube-api-access-vvpff") pod "47188b56-9ee4-4fab-99cb-6a205b2facb1" (UID: "47188b56-9ee4-4fab-99cb-6a205b2facb1"). InnerVolumeSpecName "kube-api-access-vvpff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:30:29.438299 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.438203 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47188b56-9ee4-4fab-99cb-6a205b2facb1" (UID: "47188b56-9ee4-4fab-99cb-6a205b2facb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:30:29.536793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.536724 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47188b56-9ee4-4fab-99cb-6a205b2facb1-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:30:29.536793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.536751 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47188b56-9ee4-4fab-99cb-6a205b2facb1-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:30:29.536793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.536772 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvpff\" (UniqueName: \"kubernetes.io/projected/47188b56-9ee4-4fab-99cb-6a205b2facb1-kube-api-access-vvpff\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:30:29.957450 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.957417 2570 generic.go:356] "Generic (PLEG): container finished" podID="47188b56-9ee4-4fab-99cb-6a205b2facb1" containerID="c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749" exitCode=0 Jul 21 21:30:29.957866 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.957479 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" Jul 21 21:30:29.957866 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.957493 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" event={"ID":"47188b56-9ee4-4fab-99cb-6a205b2facb1","Type":"ContainerDied","Data":"c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749"} Jul 21 21:30:29.957866 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.957547 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp" event={"ID":"47188b56-9ee4-4fab-99cb-6a205b2facb1","Type":"ContainerDied","Data":"f382c78b71582722410f12e27eda45bf82f1e552bfa87abae8eddf96fbd0448b"} Jul 21 21:30:29.957866 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.957566 2570 scope.go:117] "RemoveContainer" containerID="c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749" Jul 21 21:30:29.967166 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.967141 2570 scope.go:117] "RemoveContainer" containerID="c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749" Jul 21 21:30:29.967414 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:30:29.967396 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749\": container with ID starting with c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749 not found: ID does not exist" containerID="c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749" Jul 21 21:30:29.967478 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.967422 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749"} err="failed to get container status \"c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749\": rpc error: code = NotFound desc = could not find container \"c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749\": container with ID starting with c8ae42cdd0cb2d5622821009f3740dde7c2b45a630de5f932b7c6812ee2c3749 not found: ID does not exist" Jul 21 21:30:29.976352 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.976326 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp"] Jul 21 21:30:29.982403 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:29.982382 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e49e6eb7-577c646b8b-z9xzp"] Jul 21 21:30:31.556794 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:31.556757 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47188b56-9ee4-4fab-99cb-6a205b2facb1" path="/var/lib/kubelet/pods/47188b56-9ee4-4fab-99cb-6a205b2facb1/volumes" Jul 21 21:30:33.268411 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:33.268377 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:33.446896 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:33.446861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:33.698556 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:33.698517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:34.219527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:34.219492 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:34.315442 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:34.315401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:34.415404 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:34.415352 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:35.906633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:35.906594 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:42.123330 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:42.123298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:42.231808 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:42.231772 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:42.323950 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:42.323915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:42.426393 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:42.426346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:42.854460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:42.854354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:30:53.444433 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:30:53.444400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:08.846996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:08.846951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:15.211552 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:15.211516 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:15.273944 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:15.273915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:15.369445 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:15.369409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:15.490268 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:15.490183 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:31.364172 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:31.364113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:37.084751 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:37.084706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:37.198674 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:37.198638 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:37.296529 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:37.296494 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:47.862214 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:47.862179 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:48.680925 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:48.680874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:55.488009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:55.487975 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:55.594585 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:55.594551 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:55.691393 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:55.691334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:55.790145 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:55.790028 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:56.597955 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:56.597920 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:56.686330 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:56.686295 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:56.786376 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:56.786318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:31:58.399796 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:31:58.399757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:04.516916 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:04.516873 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:04.613092 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:04.613041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:05.192092 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:05.192055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:11.351613 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:11.351573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:11.421998 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:11.421958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:11.579113 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:11.579071 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:11.933603 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:11.933563 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:12.034812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:12.034771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:14.763592 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.763557 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:14.853848 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.853814 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85"] Jul 21 21:32:14.854257 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.854243 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47188b56-9ee4-4fab-99cb-6a205b2facb1" containerName="maas-api" Jul 21 21:32:14.854307 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.854260 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="47188b56-9ee4-4fab-99cb-6a205b2facb1" containerName="maas-api" Jul 21 21:32:14.854349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.854338 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="47188b56-9ee4-4fab-99cb-6a205b2facb1" containerName="maas-api" Jul 21 21:32:14.857507 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.857487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:14.859257 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.859227 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:14.860381 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.860335 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e31a199a-serving-cert\"" Jul 21 21:32:14.867676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.867654 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85"] Jul 21 21:32:14.932294 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.932262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:14.968129 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.968094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b370d-631b-4cf9-b190-6532fb68e2e7-maas-api-tls\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:14.968278 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.968211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgjr\" (UniqueName: \"kubernetes.io/projected/ab7b370d-631b-4cf9-b190-6532fb68e2e7-kube-api-access-sjgjr\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:14.968278 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:14.968242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b370d-631b-4cf9-b190-6532fb68e2e7-openshift-service-ca\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.030944 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.030871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:15.069126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.069088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgjr\" (UniqueName: \"kubernetes.io/projected/ab7b370d-631b-4cf9-b190-6532fb68e2e7-kube-api-access-sjgjr\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.069126 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.069129 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b370d-631b-4cf9-b190-6532fb68e2e7-openshift-service-ca\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.069337 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.069169 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b370d-631b-4cf9-b190-6532fb68e2e7-maas-api-tls\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.069891 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.069866 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b370d-631b-4cf9-b190-6532fb68e2e7-openshift-service-ca\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.071727 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.071707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b370d-631b-4cf9-b190-6532fb68e2e7-maas-api-tls\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.077351 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.077327 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgjr\" (UniqueName: \"kubernetes.io/projected/ab7b370d-631b-4cf9-b190-6532fb68e2e7-kube-api-access-sjgjr\") pod \"maas-api-e2e-mt-e31a199a-5fcd878955-gpx85\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.169815 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.169775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:15.321485 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.321459 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85"] Jul 21 21:32:15.322936 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:32:15.322902 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7b370d_631b_4cf9_b190_6532fb68e2e7.slice/crio-062e690c32c84b26e45205c5b92ae8b17ac349b6f231e4d261db5d595ac41a9a WatchSource:0}: Error finding container 062e690c32c84b26e45205c5b92ae8b17ac349b6f231e4d261db5d595ac41a9a: Status 404 returned error can't find the container with id 062e690c32c84b26e45205c5b92ae8b17ac349b6f231e4d261db5d595ac41a9a Jul 21 21:32:15.460863 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.460826 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:15.546681 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.546646 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" event={"ID":"ab7b370d-631b-4cf9-b190-6532fb68e2e7","Type":"ContainerStarted","Data":"062e690c32c84b26e45205c5b92ae8b17ac349b6f231e4d261db5d595ac41a9a"} Jul 21 21:32:15.564419 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:15.564332 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:16.560768 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:16.560726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" event={"ID":"ab7b370d-631b-4cf9-b190-6532fb68e2e7","Type":"ContainerStarted","Data":"782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab"} Jul 21 21:32:16.561231 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:16.560820 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:16.581161 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:16.581104 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" podStartSLOduration=2.161646574 podStartE2EDuration="2.581089234s" totalImagesPullingTime="419.44266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:32:16.579670205 +0000 UTC m=+1913.524291302" watchObservedRunningTime="2026-07-21 21:32:16.581089234 +0000 UTC m=+1913.525710343" Jul 21 21:32:19.303344 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:19.303298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:20.033984 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:20.033948 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:20.143028 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:20.142995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:20.479869 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:20.479833 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:22.572115 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:22.572082 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:26.251280 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:26.251242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:26.315266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:26.315234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:37.344838 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.344806 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85"] Jul 21 21:32:37.345567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.345531 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" podUID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" containerName="maas-api" containerID="cri-o://782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab" gracePeriod=30 Jul 21 21:32:37.427629 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.427592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:37.605870 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.605814 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:37.672092 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.672060 2570 generic.go:356] "Generic (PLEG): container finished" podID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" containerID="782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab" exitCode=0 Jul 21 21:32:37.672248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.672104 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" Jul 21 21:32:37.672248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.672141 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" event={"ID":"ab7b370d-631b-4cf9-b190-6532fb68e2e7","Type":"ContainerDied","Data":"782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab"} Jul 21 21:32:37.672248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.672191 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" event={"ID":"ab7b370d-631b-4cf9-b190-6532fb68e2e7","Type":"ContainerDied","Data":"062e690c32c84b26e45205c5b92ae8b17ac349b6f231e4d261db5d595ac41a9a"} Jul 21 21:32:37.672248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.672215 2570 scope.go:117] "RemoveContainer" containerID="782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab" Jul 21 21:32:37.682210 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.682189 2570 scope.go:117] "RemoveContainer" containerID="782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab" Jul 21 21:32:37.682271 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.682220 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjgjr\" (UniqueName: \"kubernetes.io/projected/ab7b370d-631b-4cf9-b190-6532fb68e2e7-kube-api-access-sjgjr\") pod \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " Jul 21 21:32:37.682318 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.682276 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b370d-631b-4cf9-b190-6532fb68e2e7-openshift-service-ca\") pod \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " Jul 21 21:32:37.682400 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.682349 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b370d-631b-4cf9-b190-6532fb68e2e7-maas-api-tls\") pod \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\" (UID: \"ab7b370d-631b-4cf9-b190-6532fb68e2e7\") " Jul 21 21:32:37.682487 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:32:37.682466 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab\": container with ID starting with 782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab not found: ID does not exist" containerID="782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab" Jul 21 21:32:37.682551 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.682497 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab"} err="failed to get container status \"782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab\": rpc error: code = NotFound desc = could not find container \"782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab\": container with ID starting with 782e96a5ea5c81ba94fc2ea761fb7bb8996ddbfc9765d8c01fc88f730c560eab not found: ID does not exist" Jul 21 21:32:37.682689 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.682669 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7b370d-631b-4cf9-b190-6532fb68e2e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab7b370d-631b-4cf9-b190-6532fb68e2e7" (UID: "ab7b370d-631b-4cf9-b190-6532fb68e2e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:32:37.684460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.684441 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7b370d-631b-4cf9-b190-6532fb68e2e7-kube-api-access-sjgjr" (OuterVolumeSpecName: "kube-api-access-sjgjr") pod "ab7b370d-631b-4cf9-b190-6532fb68e2e7" (UID: "ab7b370d-631b-4cf9-b190-6532fb68e2e7"). InnerVolumeSpecName "kube-api-access-sjgjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:32:37.684521 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.684489 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7b370d-631b-4cf9-b190-6532fb68e2e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab7b370d-631b-4cf9-b190-6532fb68e2e7" (UID: "ab7b370d-631b-4cf9-b190-6532fb68e2e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:32:37.783430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.783406 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b370d-631b-4cf9-b190-6532fb68e2e7-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:32:37.783430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.783427 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b370d-631b-4cf9-b190-6532fb68e2e7-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:32:37.783571 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.783437 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgjr\" (UniqueName: \"kubernetes.io/projected/ab7b370d-631b-4cf9-b190-6532fb68e2e7-kube-api-access-sjgjr\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:32:37.997904 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:37.997873 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85"] Jul 21 21:32:38.001923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:38.001899 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85"] Jul 21 21:32:39.556827 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:39.556791 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" path="/var/lib/kubelet/pods/ab7b370d-631b-4cf9-b190-6532fb68e2e7/volumes" Jul 21 21:32:40.568111 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:40.568068 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e31a199a-5fcd878955-gpx85" podUID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.58:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 21 21:32:41.797054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:41.797016 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:42.223634 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:42.223587 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:48.444996 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:48.444959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:48.547937 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:48.547899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:48.748237 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:48.748209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:48.960577 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:48.960538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:49.096216 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:49.096120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:49.186061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:49.186021 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:49.325561 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:49.325524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:55.359099 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:55.359066 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:55.464710 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:55.464636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:55.567041 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:55.567011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:32:55.737618 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:32:55.737580 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:01.495909 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:01.495872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:01.579280 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:01.579248 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:08.422422 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:08.422382 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:08.790180 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:08.790096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:09.312041 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:09.311995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:11.950004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:11.949967 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:12.086887 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.086855 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:12.173506 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.173475 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm"] Jul 21 21:33:12.173971 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.173954 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" containerName="maas-api" Jul 21 21:33:12.174064 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.173974 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" containerName="maas-api" Jul 21 21:33:12.174064 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.174057 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7b370d-631b-4cf9-b190-6532fb68e2e7" containerName="maas-api" Jul 21 21:33:12.176992 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.176971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.180610 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.180589 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ebe935a0-serving-cert\"" Jul 21 21:33:12.212145 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.212088 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm"] Jul 21 21:33:12.284005 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.283970 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghn4\" (UniqueName: \"kubernetes.io/projected/eaabbb6b-253f-4b1c-a5a2-9db116521076-kube-api-access-tghn4\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.284194 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.284016 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.284194 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.284101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaabbb6b-253f-4b1c-a5a2-9db116521076-openshift-service-ca\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.291402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.291349 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:12.384641 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.384609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghn4\" (UniqueName: \"kubernetes.io/projected/eaabbb6b-253f-4b1c-a5a2-9db116521076-kube-api-access-tghn4\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.384641 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.384642 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.384842 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.384687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaabbb6b-253f-4b1c-a5a2-9db116521076-openshift-service-ca\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.384975 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:33:12.384954 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-serving-cert: secret "maas-api-e2e-mt-ebe935a0-serving-cert" not found Jul 21 21:33:12.385097 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:33:12.385043 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls podName:eaabbb6b-253f-4b1c-a5a2-9db116521076 nodeName:}" failed. No retries permitted until 2026-07-21 21:33:12.885020761 +0000 UTC m=+1969.829641811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls") pod "maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" (UID: "eaabbb6b-253f-4b1c-a5a2-9db116521076") : secret "maas-api-e2e-mt-ebe935a0-serving-cert" not found Jul 21 21:33:12.385334 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.385316 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaabbb6b-253f-4b1c-a5a2-9db116521076-openshift-service-ca\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.405961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.405935 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghn4\" (UniqueName: \"kubernetes.io/projected/eaabbb6b-253f-4b1c-a5a2-9db116521076-kube-api-access-tghn4\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.552453 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.552351 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:12.890477 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.890394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:12.892918 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:12.892900 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls\") pod \"maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:13.070713 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.070681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:13.087646 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.087617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:13.176956 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.176899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:13.252183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.252152 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm"] Jul 21 21:33:13.254333 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:33:13.254301 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaabbb6b_253f_4b1c_a5a2_9db116521076.slice/crio-27e06a04560fe04e6c3fd1f989fd393ecae8e905aaa8680f621f15e7a59a1db0 WatchSource:0}: Error finding container 27e06a04560fe04e6c3fd1f989fd393ecae8e905aaa8680f621f15e7a59a1db0: Status 404 returned error can't find the container with id 27e06a04560fe04e6c3fd1f989fd393ecae8e905aaa8680f621f15e7a59a1db0 Jul 21 21:33:13.255805 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.255782 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:33:13.876831 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.876740 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" event={"ID":"eaabbb6b-253f-4b1c-a5a2-9db116521076","Type":"ContainerStarted","Data":"9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147"} Jul 21 21:33:13.877010 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.876836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" event={"ID":"eaabbb6b-253f-4b1c-a5a2-9db116521076","Type":"ContainerStarted","Data":"27e06a04560fe04e6c3fd1f989fd393ecae8e905aaa8680f621f15e7a59a1db0"} Jul 21 21:33:13.877010 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.876875 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:13.910413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:13.910323 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" podStartSLOduration=1.569064516 podStartE2EDuration="1.910305652s" totalImagesPullingTime="341.241136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:33:13.909640127 +0000 UTC m=+1970.854261214" watchObservedRunningTime="2026-07-21 21:33:13.910305652 +0000 UTC m=+1970.854926718" Jul 21 21:33:16.445111 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:16.445031 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:33:19.888512 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:19.888480 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:33:27.551401 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:33:27.551350 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:05.495430 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.495337 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm"] Jul 21 21:34:05.496540 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.496481 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" podUID="eaabbb6b-253f-4b1c-a5a2-9db116521076" containerName="maas-api" containerID="cri-o://9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147" gracePeriod=30 Jul 21 21:34:05.521291 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.521261 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:05.756187 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.756126 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:34:05.853248 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.853218 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls\") pod \"eaabbb6b-253f-4b1c-a5a2-9db116521076\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " Jul 21 21:34:05.853469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.853284 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tghn4\" (UniqueName: \"kubernetes.io/projected/eaabbb6b-253f-4b1c-a5a2-9db116521076-kube-api-access-tghn4\") pod \"eaabbb6b-253f-4b1c-a5a2-9db116521076\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " Jul 21 21:34:05.853469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.853328 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaabbb6b-253f-4b1c-a5a2-9db116521076-openshift-service-ca\") pod \"eaabbb6b-253f-4b1c-a5a2-9db116521076\" (UID: \"eaabbb6b-253f-4b1c-a5a2-9db116521076\") " Jul 21 21:34:05.853775 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.853748 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaabbb6b-253f-4b1c-a5a2-9db116521076-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaabbb6b-253f-4b1c-a5a2-9db116521076" (UID: "eaabbb6b-253f-4b1c-a5a2-9db116521076"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:34:05.855441 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.855408 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaabbb6b-253f-4b1c-a5a2-9db116521076-kube-api-access-tghn4" (OuterVolumeSpecName: "kube-api-access-tghn4") pod "eaabbb6b-253f-4b1c-a5a2-9db116521076" (UID: "eaabbb6b-253f-4b1c-a5a2-9db116521076"). InnerVolumeSpecName "kube-api-access-tghn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:34:05.855441 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.855420 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaabbb6b-253f-4b1c-a5a2-9db116521076" (UID: "eaabbb6b-253f-4b1c-a5a2-9db116521076"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:34:05.954968 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.954928 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaabbb6b-253f-4b1c-a5a2-9db116521076-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:34:05.954968 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.954959 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghn4\" (UniqueName: \"kubernetes.io/projected/eaabbb6b-253f-4b1c-a5a2-9db116521076-kube-api-access-tghn4\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:34:05.954968 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:05.954970 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaabbb6b-253f-4b1c-a5a2-9db116521076-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:34:06.178625 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.178595 2570 generic.go:356] "Generic (PLEG): container finished" podID="eaabbb6b-253f-4b1c-a5a2-9db116521076" containerID="9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147" exitCode=0 Jul 21 21:34:06.178798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.178651 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" Jul 21 21:34:06.178798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.178682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" event={"ID":"eaabbb6b-253f-4b1c-a5a2-9db116521076","Type":"ContainerDied","Data":"9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147"} Jul 21 21:34:06.178798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.178759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm" event={"ID":"eaabbb6b-253f-4b1c-a5a2-9db116521076","Type":"ContainerDied","Data":"27e06a04560fe04e6c3fd1f989fd393ecae8e905aaa8680f621f15e7a59a1db0"} Jul 21 21:34:06.178798 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.178784 2570 scope.go:117] "RemoveContainer" containerID="9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147" Jul 21 21:34:06.188112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.188094 2570 scope.go:117] "RemoveContainer" containerID="9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147" Jul 21 21:34:06.188352 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:34:06.188336 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147\": container with ID starting with 9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147 not found: ID does not exist" containerID="9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147" Jul 21 21:34:06.188434 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.188376 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147"} err="failed to get container status \"9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147\": rpc error: code = NotFound desc = could not find container \"9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147\": container with ID starting with 9edb13569a94b2d1c0030ffa51d122444c5ac9a1c2779bfddb671d9ce14bf147 not found: ID does not exist" Jul 21 21:34:06.208870 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.208836 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm"] Jul 21 21:34:06.212941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:06.212920 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebe935a0-75dcb8c9cc-w9rfm"] Jul 21 21:34:07.557488 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:07.557454 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaabbb6b-253f-4b1c-a5a2-9db116521076" path="/var/lib/kubelet/pods/eaabbb6b-253f-4b1c-a5a2-9db116521076/volumes" Jul 21 21:34:10.065649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:10.065611 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:12.351439 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:12.351400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:17.427428 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:17.427392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:45.792786 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:45.792703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:54.309302 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:54.309248 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:34:54.844569 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:34:54.844536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:00.527955 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:00.527921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:00.597004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:00.596962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:00.822416 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:00.822307 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:01.081024 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:01.080939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:02.230768 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:02.230734 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:02.350627 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:02.350595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:03.147978 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:03.147921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:03.404334 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:03.404208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:03.547107 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:03.544644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:04.844462 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:04.844422 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:13.003058 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:13.003022 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:13.153986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:13.153950 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:13.807898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:13.807851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:20.174275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:20.174242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:20.301123 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:20.301087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:20.397528 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:20.397494 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:22.016411 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:22.016356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:22.116867 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:22.116820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:30.914958 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:30.914911 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:31.249880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:31.249796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:36.595040 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:36.595007 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:44.947636 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:44.947600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:52.232446 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:52.232390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:58.816857 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:58.816826 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:58.917549 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:58.917515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:59.168141 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:59.168114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:59.487112 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:59.487035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:35:59.856994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:35:59.856912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:02.822418 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.822378 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:02.927542 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.927504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:02.949838 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.949801 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4"] Jul 21 21:36:02.950219 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.950206 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaabbb6b-253f-4b1c-a5a2-9db116521076" containerName="maas-api" Jul 21 21:36:02.950270 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.950221 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaabbb6b-253f-4b1c-a5a2-9db116521076" containerName="maas-api" Jul 21 21:36:02.950309 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.950285 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaabbb6b-253f-4b1c-a5a2-9db116521076" containerName="maas-api" Jul 21 21:36:02.953210 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.953194 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:02.956462 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.956437 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bacc91-serving-cert\"" Jul 21 21:36:02.964022 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:02.963993 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4"] Jul 21 21:36:03.003806 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.003771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a8b792-db9b-441a-87c3-3bc824f8febb-maas-api-tls\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.003985 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.003849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a8b792-db9b-441a-87c3-3bc824f8febb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.003985 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.003870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpdh\" (UniqueName: \"kubernetes.io/projected/d3a8b792-db9b-441a-87c3-3bc824f8febb-kube-api-access-qqpdh\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.029557 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.029513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:03.104567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.104483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a8b792-db9b-441a-87c3-3bc824f8febb-maas-api-tls\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.104715 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.104579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a8b792-db9b-441a-87c3-3bc824f8febb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.104715 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.104604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpdh\" (UniqueName: \"kubernetes.io/projected/d3a8b792-db9b-441a-87c3-3bc824f8febb-kube-api-access-qqpdh\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.105326 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.105305 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a8b792-db9b-441a-87c3-3bc824f8febb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.107343 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.107319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a8b792-db9b-441a-87c3-3bc824f8febb-maas-api-tls\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.119061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.119032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpdh\" (UniqueName: \"kubernetes.io/projected/d3a8b792-db9b-441a-87c3-3bc824f8febb-kube-api-access-qqpdh\") pod \"maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.168429 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.168391 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:03.264357 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.264302 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:03.411308 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.411277 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4"] Jul 21 21:36:03.412539 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:36:03.412508 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a8b792_db9b_441a_87c3_3bc824f8febb.slice/crio-a139d0d4319744db2704a908430744153993e64450066895bef25907eb2173ca WatchSource:0}: Error finding container a139d0d4319744db2704a908430744153993e64450066895bef25907eb2173ca: Status 404 returned error can't find the container with id a139d0d4319744db2704a908430744153993e64450066895bef25907eb2173ca Jul 21 21:36:03.579900 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.579862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:03.692495 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.692455 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:03.841725 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:03.841690 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" event={"ID":"d3a8b792-db9b-441a-87c3-3bc824f8febb","Type":"ContainerStarted","Data":"a139d0d4319744db2704a908430744153993e64450066895bef25907eb2173ca"} Jul 21 21:36:04.849124 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:04.849089 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" event={"ID":"d3a8b792-db9b-441a-87c3-3bc824f8febb","Type":"ContainerStarted","Data":"5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b"} Jul 21 21:36:04.849546 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:04.849220 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:04.868531 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:04.868480 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" podStartSLOduration=2.30799962 podStartE2EDuration="2.868463364s" totalImagesPullingTime="560.463744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:36:04.866292354 +0000 UTC m=+2141.810913460" watchObservedRunningTime="2026-07-21 21:36:04.868463364 +0000 UTC m=+2141.813084445" Jul 21 21:36:07.329087 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:07.329055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:07.656249 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:07.656164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:07.990703 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:07.990673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:08.084312 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:08.084273 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:10.783354 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:10.783309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:10.859667 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:10.859637 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:36:10.886812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:10.886783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:11.047889 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.047797 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277"] Jul 21 21:36:11.051784 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.051761 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.054490 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.054470 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9e03c3-serving-cert\"" Jul 21 21:36:11.061786 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.061763 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277"] Jul 21 21:36:11.074840 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.074809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.074989 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.074903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4wk\" (UniqueName: \"kubernetes.io/projected/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-kube-api-access-zh4wk\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.075044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.075013 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.081892 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.081861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:11.176235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.176200 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.176455 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.176270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4wk\" (UniqueName: \"kubernetes.io/projected/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-kube-api-access-zh4wk\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.176455 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.176331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.176455 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:36:11.176398 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-serving-cert: secret "maas-api-e2e-inf-b-9e03c3-serving-cert" not found Jul 21 21:36:11.176588 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:36:11.176493 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls podName:d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3 nodeName:}" failed. No retries permitted until 2026-07-21 21:36:11.676470703 +0000 UTC m=+2148.621091746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls") pod "maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" (UID: "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3") : secret "maas-api-e2e-inf-b-9e03c3-serving-cert" not found Jul 21 21:36:11.176951 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.176934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.189698 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.189670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4wk\" (UniqueName: \"kubernetes.io/projected/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-kube-api-access-zh4wk\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.399104 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.399027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:11.603902 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.603870 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:11.680343 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.680306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.682952 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.682929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:11.705973 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.705944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:11.965179 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:11.965088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:12.119047 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:12.119011 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277"] Jul 21 21:36:12.900935 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:12.900832 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" event={"ID":"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3","Type":"ContainerStarted","Data":"7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6"} Jul 21 21:36:12.900935 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:12.900913 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:12.901106 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:12.900938 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" event={"ID":"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3","Type":"ContainerStarted","Data":"6dbc3780443eb69091669d6a9e6764d690f8876f5a5ebdd8d6b45c3b228113b5"} Jul 21 21:36:12.921164 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:12.921106 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" podStartSLOduration=1.528679095 podStartE2EDuration="1.921091899s" totalImagesPullingTime="392.412804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:36:12.919527777 +0000 UTC m=+2149.864148842" watchObservedRunningTime="2026-07-21 21:36:12.921091899 +0000 UTC m=+2149.865712964" Jul 21 21:36:15.302084 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:15.302000 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:18.561885 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:18.561854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:18.911016 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:18.910940 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:36:24.512664 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:24.512604 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:24.606043 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:24.606000 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:32.424594 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:32.424556 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:36:32.527024 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:36:32.526981 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:05.090104 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:05.090064 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:45.300349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:45.300246 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:46.349469 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.349428 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb"] Jul 21 21:37:46.352945 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.352927 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:46.355745 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.355722 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:37:46.361322 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.361300 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb"] Jul 21 21:37:46.433904 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.433871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb\" (UID: \"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:46.526519 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.526486 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:46.534756 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.534720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb\" (UID: \"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:46.535405 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.535351 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb\" (UID: \"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:46.665769 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.665737 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:46.810035 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:37:46.810006 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb190b56f_6ae7_4598_9c19_b50fbf5cb4fa.slice/crio-c8b12c47d8c06779a176138fe7e2be78afff3106b2e443833f0ce87fbd7e5a54 WatchSource:0}: Error finding container c8b12c47d8c06779a176138fe7e2be78afff3106b2e443833f0ce87fbd7e5a54: Status 404 returned error can't find the container with id c8b12c47d8c06779a176138fe7e2be78afff3106b2e443833f0ce87fbd7e5a54 Jul 21 21:37:46.810536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.810507 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb"] Jul 21 21:37:46.918729 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:46.918648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:47.021334 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:47.021302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:47.415730 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:47.415699 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" event={"ID":"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa","Type":"ContainerStarted","Data":"c8b12c47d8c06779a176138fe7e2be78afff3106b2e443833f0ce87fbd7e5a54"} Jul 21 21:37:50.436376 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:50.436331 2570 generic.go:356] "Generic (PLEG): container finished" podID="b190b56f-6ae7-4598-9c19-b50fbf5cb4fa" containerID="c6b515a90762f11b3daae7732382bfa3b406da5eaa8bca30e43ee423dc5b2250" exitCode=0 Jul 21 21:37:50.436747 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:50.436418 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" event={"ID":"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa","Type":"ContainerDied","Data":"c6b515a90762f11b3daae7732382bfa3b406da5eaa8bca30e43ee423dc5b2250"} Jul 21 21:37:51.584809 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:51.584784 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:51.682917 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:51.682887 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa-maas-api-service-ca\") pod \"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa\" (UID: \"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa\") " Jul 21 21:37:51.683221 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:51.683199 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b190b56f-6ae7-4598-9c19-b50fbf5cb4fa" (UID: "b190b56f-6ae7-4598-9c19-b50fbf5cb4fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:37:51.783761 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:51.783697 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa-maas-api-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:37:52.454630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:52.451436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" event={"ID":"b190b56f-6ae7-4598-9c19-b50fbf5cb4fa","Type":"ContainerDied","Data":"c8b12c47d8c06779a176138fe7e2be78afff3106b2e443833f0ce87fbd7e5a54"} Jul 21 21:37:52.454630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:52.451543 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b12c47d8c06779a176138fe7e2be78afff3106b2e443833f0ce87fbd7e5a54" Jul 21 21:37:52.454630 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:52.451623 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb" Jul 21 21:37:56.785376 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:56.785323 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4"] Jul 21 21:37:56.785760 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:56.785597 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" podUID="d3a8b792-db9b-441a-87c3-3bc824f8febb" containerName="maas-api" containerID="cri-o://5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b" gracePeriod=30 Jul 21 21:37:56.830514 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:56.830485 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:37:57.042704 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.042638 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:37:57.130574 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.130540 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqpdh\" (UniqueName: \"kubernetes.io/projected/d3a8b792-db9b-441a-87c3-3bc824f8febb-kube-api-access-qqpdh\") pod \"d3a8b792-db9b-441a-87c3-3bc824f8febb\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " Jul 21 21:37:57.130574 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.130579 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a8b792-db9b-441a-87c3-3bc824f8febb-maas-api-tls\") pod \"d3a8b792-db9b-441a-87c3-3bc824f8febb\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " Jul 21 21:37:57.130778 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.130602 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a8b792-db9b-441a-87c3-3bc824f8febb-openshift-service-ca\") pod \"d3a8b792-db9b-441a-87c3-3bc824f8febb\" (UID: \"d3a8b792-db9b-441a-87c3-3bc824f8febb\") " Jul 21 21:37:57.131047 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.131022 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a8b792-db9b-441a-87c3-3bc824f8febb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3a8b792-db9b-441a-87c3-3bc824f8febb" (UID: "d3a8b792-db9b-441a-87c3-3bc824f8febb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:37:57.132965 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.132933 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a8b792-db9b-441a-87c3-3bc824f8febb-kube-api-access-qqpdh" (OuterVolumeSpecName: "kube-api-access-qqpdh") pod "d3a8b792-db9b-441a-87c3-3bc824f8febb" (UID: "d3a8b792-db9b-441a-87c3-3bc824f8febb"). InnerVolumeSpecName "kube-api-access-qqpdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:37:57.133085 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.132985 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a8b792-db9b-441a-87c3-3bc824f8febb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3a8b792-db9b-441a-87c3-3bc824f8febb" (UID: "d3a8b792-db9b-441a-87c3-3bc824f8febb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:37:57.232147 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.232109 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqpdh\" (UniqueName: \"kubernetes.io/projected/d3a8b792-db9b-441a-87c3-3bc824f8febb-kube-api-access-qqpdh\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:37:57.232147 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.232143 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a8b792-db9b-441a-87c3-3bc824f8febb-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:37:57.232346 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.232158 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a8b792-db9b-441a-87c3-3bc824f8febb-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:37:57.481494 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.481462 2570 generic.go:356] "Generic (PLEG): container finished" podID="d3a8b792-db9b-441a-87c3-3bc824f8febb" containerID="5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b" exitCode=0 Jul 21 21:37:57.481676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.481511 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" Jul 21 21:37:57.481676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.481551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" event={"ID":"d3a8b792-db9b-441a-87c3-3bc824f8febb","Type":"ContainerDied","Data":"5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b"} Jul 21 21:37:57.481676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.481603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4" event={"ID":"d3a8b792-db9b-441a-87c3-3bc824f8febb","Type":"ContainerDied","Data":"a139d0d4319744db2704a908430744153993e64450066895bef25907eb2173ca"} Jul 21 21:37:57.481676 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.481635 2570 scope.go:117] "RemoveContainer" containerID="5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b" Jul 21 21:37:57.491665 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.491649 2570 scope.go:117] "RemoveContainer" containerID="5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b" Jul 21 21:37:57.491939 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:37:57.491918 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b\": container with ID starting with 5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b not found: ID does not exist" containerID="5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b" Jul 21 21:37:57.492015 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.491947 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b"} err="failed to get container status \"5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b\": rpc error: code = NotFound desc = could not find container \"5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b\": container with ID starting with 5053ac10e1be6e38a24dc818711a09a3e4c1af035f1911f6f65ff8a80c12f05b not found: ID does not exist" Jul 21 21:37:57.505239 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.505213 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4"] Jul 21 21:37:57.509116 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.509092 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bacc91-7f9799bc7d-gp8j4"] Jul 21 21:37:57.557924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:37:57.557894 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a8b792-db9b-441a-87c3-3bc824f8febb" path="/var/lib/kubelet/pods/d3a8b792-db9b-441a-87c3-3bc824f8febb/volumes" Jul 21 21:38:00.981178 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:00.981141 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:01.052591 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:01.052561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:01.438958 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:01.438920 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb"] Jul 21 21:38:01.441235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:01.441211 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bacc91-qlsmb"] Jul 21 21:38:01.445649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:01.445624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:01.556623 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:01.556590 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b190b56f-6ae7-4598-9c19-b50fbf5cb4fa" path="/var/lib/kubelet/pods/b190b56f-6ae7-4598-9c19-b50fbf5cb4fa/volumes" Jul 21 21:38:01.840588 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:01.840509 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:07.869073 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:07.869037 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:07.967802 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:07.967756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:08.145592 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:08.145486 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:08.375920 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:08.375883 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:09.178621 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:09.178583 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:09.295037 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:09.295000 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:19.101718 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.101673 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277"] Jul 21 21:38:19.102334 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.101967 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" podUID="d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" containerName="maas-api" containerID="cri-o://7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6" gracePeriod=30 Jul 21 21:38:19.164621 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.164575 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:19.353970 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.353918 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:38:19.533420 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.533387 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls\") pod \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " Jul 21 21:38:19.533616 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.533430 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh4wk\" (UniqueName: \"kubernetes.io/projected/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-kube-api-access-zh4wk\") pod \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " Jul 21 21:38:19.533616 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.533564 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-openshift-service-ca\") pod \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\" (UID: \"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3\") " Jul 21 21:38:19.533943 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.533920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" (UID: "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:38:19.535693 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.535657 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" (UID: "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:38:19.535838 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.535811 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-kube-api-access-zh4wk" (OuterVolumeSpecName: "kube-api-access-zh4wk") pod "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" (UID: "d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3"). InnerVolumeSpecName "kube-api-access-zh4wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:38:19.599682 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.599648 2570 generic.go:356] "Generic (PLEG): container finished" podID="d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" containerID="7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6" exitCode=0 Jul 21 21:38:19.599812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.599706 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" Jul 21 21:38:19.599812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.599735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" event={"ID":"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3","Type":"ContainerDied","Data":"7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6"} Jul 21 21:38:19.599812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.599785 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277" event={"ID":"d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3","Type":"ContainerDied","Data":"6dbc3780443eb69091669d6a9e6764d690f8876f5a5ebdd8d6b45c3b228113b5"} Jul 21 21:38:19.599812 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.599807 2570 scope.go:117] "RemoveContainer" containerID="7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6" Jul 21 21:38:19.612255 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.612236 2570 scope.go:117] "RemoveContainer" containerID="7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6" Jul 21 21:38:19.612547 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:38:19.612527 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6\": container with ID starting with 7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6 not found: ID does not exist" containerID="7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6" Jul 21 21:38:19.612626 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.612554 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6"} err="failed to get container status \"7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6\": rpc error: code = NotFound desc = could not find container \"7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6\": container with ID starting with 7fef70ddc4382e30569bfb6fa71453b29e9f947b39350aa3327e66cb1e239ef6 not found: ID does not exist" Jul 21 21:38:19.618227 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.618202 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277"] Jul 21 21:38:19.622819 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.622799 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9e03c3-5967bc978f-fp277"] Jul 21 21:38:19.635128 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.635111 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:38:19.635207 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.635130 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:38:19.635207 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:19.635140 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh4wk\" (UniqueName: \"kubernetes.io/projected/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3-kube-api-access-zh4wk\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:38:21.557266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:21.557231 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" path="/var/lib/kubelet/pods/d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3/volumes" Jul 21 21:38:23.579199 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:23.579166 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:23.987898 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:23.987856 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:30.141157 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:30.141120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:30.216816 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:30.216783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:30.426030 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:30.426002 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:30.654890 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:30.654854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:31.038848 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:31.038814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:31.384708 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:31.384619 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:31.674349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:31.674306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:34.583660 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.583626 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:34.687387 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.687334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:34.720402 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720356 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7"] Jul 21 21:38:34.720796 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720783 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a8b792-db9b-441a-87c3-3bc824f8febb" containerName="maas-api" Jul 21 21:38:34.720796 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720797 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a8b792-db9b-441a-87c3-3bc824f8febb" containerName="maas-api" Jul 21 21:38:34.720881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720814 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" containerName="maas-api" Jul 21 21:38:34.720881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720820 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" containerName="maas-api" Jul 21 21:38:34.720881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720828 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b190b56f-6ae7-4598-9c19-b50fbf5cb4fa" containerName="revoke-keys" Jul 21 21:38:34.720881 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720835 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b190b56f-6ae7-4598-9c19-b50fbf5cb4fa" containerName="revoke-keys" Jul 21 21:38:34.721009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720904 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15fad2c-073d-4df4-a8d6-bcfb35b0f8a3" containerName="maas-api" Jul 21 21:38:34.721009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720915 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b190b56f-6ae7-4598-9c19-b50fbf5cb4fa" containerName="revoke-keys" Jul 21 21:38:34.721009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.720923 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a8b792-db9b-441a-87c3-3bc824f8febb" containerName="maas-api" Jul 21 21:38:34.724038 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.724018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.727021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.727002 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-226c3e-serving-cert\"" Jul 21 21:38:34.738178 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.738155 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7"] Jul 21 21:38:34.768044 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.768014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.768198 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.768069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f78b72-a6de-4eba-a1e3-b228db477b40-openshift-service-ca\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.768198 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.768088 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvwm\" (UniqueName: \"kubernetes.io/projected/10f78b72-a6de-4eba-a1e3-b228db477b40-kube-api-access-hbvwm\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.783653 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.783623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:34.869607 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.869514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.869754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.869609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f78b72-a6de-4eba-a1e3-b228db477b40-openshift-service-ca\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.869754 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:38:34.869634 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-serving-cert: secret "maas-api-e2e-api-a-226c3e-serving-cert" not found Jul 21 21:38:34.869754 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:38:34.869732 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls podName:10f78b72-a6de-4eba-a1e3-b228db477b40 nodeName:}" failed. No retries permitted until 2026-07-21 21:38:35.369711298 +0000 UTC m=+2292.314332362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls") pod "maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" (UID: "10f78b72-a6de-4eba-a1e3-b228db477b40") : secret "maas-api-e2e-api-a-226c3e-serving-cert" not found Jul 21 21:38:34.869867 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.869635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvwm\" (UniqueName: \"kubernetes.io/projected/10f78b72-a6de-4eba-a1e3-b228db477b40-kube-api-access-hbvwm\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.870235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.870216 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f78b72-a6de-4eba-a1e3-b228db477b40-openshift-service-ca\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:34.880398 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:34.880352 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvwm\" (UniqueName: \"kubernetes.io/projected/10f78b72-a6de-4eba-a1e3-b228db477b40-kube-api-access-hbvwm\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:35.031488 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.031459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:35.374916 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.374878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:35.377621 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.377592 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls\") pod \"maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:35.557567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.557531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:35.634509 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.634421 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:35.647977 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.647950 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:35.781280 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.781259 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7"] Jul 21 21:38:35.783176 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:38:35.783145 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f78b72_a6de_4eba_a1e3_b228db477b40.slice/crio-8a073379a784547953ac733ba785e96edcb83cec11e73c73e5acf57c40f3e1ab WatchSource:0}: Error finding container 8a073379a784547953ac733ba785e96edcb83cec11e73c73e5acf57c40f3e1ab: Status 404 returned error can't find the container with id 8a073379a784547953ac733ba785e96edcb83cec11e73c73e5acf57c40f3e1ab Jul 21 21:38:35.784553 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:35.784532 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:38:36.701225 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:36.701193 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" event={"ID":"10f78b72-a6de-4eba-a1e3-b228db477b40","Type":"ContainerStarted","Data":"4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0"} Jul 21 21:38:36.701668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:36.701267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" event={"ID":"10f78b72-a6de-4eba-a1e3-b228db477b40","Type":"ContainerStarted","Data":"8a073379a784547953ac733ba785e96edcb83cec11e73c73e5acf57c40f3e1ab"} Jul 21 21:38:36.701668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:36.701323 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:36.719937 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:36.719883 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" podStartSLOduration=2.258674469 podStartE2EDuration="2.719865513s" totalImagesPullingTime="461.191044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:38:36.717697663 +0000 UTC m=+2293.662318728" watchObservedRunningTime="2026-07-21 21:38:36.719865513 +0000 UTC m=+2293.664486579" Jul 21 21:38:39.104876 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:39.104842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:39.585935 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:39.585899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:39.934964 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:39.934930 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:40.391490 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:40.391408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:42.635408 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.635346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:42.715431 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.715401 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:38:42.735656 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.735619 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:42.763788 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.763672 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl"] Jul 21 21:38:42.768080 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.768057 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.770896 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.770878 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5898c4-serving-cert\"" Jul 21 21:38:42.782637 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.782610 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl"] Jul 21 21:38:42.841100 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.841070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9v2\" (UniqueName: \"kubernetes.io/projected/65409ab7-338e-444e-8ba3-16661ce05bf5-kube-api-access-wq9v2\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.841241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.841152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.841241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.841198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65409ab7-338e-444e-8ba3-16661ce05bf5-openshift-service-ca\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.842880 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.842847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:42.941825 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.941796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.942004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.941855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65409ab7-338e-444e-8ba3-16661ce05bf5-openshift-service-ca\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.942004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.941878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9v2\" (UniqueName: \"kubernetes.io/projected/65409ab7-338e-444e-8ba3-16661ce05bf5-kube-api-access-wq9v2\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.942004 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:38:42.941978 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-serving-cert: secret "maas-api-e2e-api-b-5898c4-serving-cert" not found Jul 21 21:38:42.942111 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:38:42.942068 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls podName:65409ab7-338e-444e-8ba3-16661ce05bf5 nodeName:}" failed. No retries permitted until 2026-07-21 21:38:43.442044721 +0000 UTC m=+2300.386665767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls") pod "maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" (UID: "65409ab7-338e-444e-8ba3-16661ce05bf5") : secret "maas-api-e2e-api-b-5898c4-serving-cert" not found Jul 21 21:38:42.942616 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.942600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65409ab7-338e-444e-8ba3-16661ce05bf5-openshift-service-ca\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:42.960674 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:42.960644 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9v2\" (UniqueName: \"kubernetes.io/projected/65409ab7-338e-444e-8ba3-16661ce05bf5-kube-api-access-wq9v2\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:43.317043 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.316960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:43.416544 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.416512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:43.447941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.447914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:43.450413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.450392 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls\") pod \"maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:43.517594 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.517557 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:43.683256 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.683217 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:43.828387 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:43.828343 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl"] Jul 21 21:38:43.831444 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:38:43.831410 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65409ab7_338e_444e_8ba3_16661ce05bf5.slice/crio-c0cb5f5360a9f28e862bb264fa0bf458e9b3e3fc7795e21ff085a9d43ee5f9fa WatchSource:0}: Error finding container c0cb5f5360a9f28e862bb264fa0bf458e9b3e3fc7795e21ff085a9d43ee5f9fa: Status 404 returned error can't find the container with id c0cb5f5360a9f28e862bb264fa0bf458e9b3e3fc7795e21ff085a9d43ee5f9fa Jul 21 21:38:44.751341 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:44.751298 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" event={"ID":"65409ab7-338e-444e-8ba3-16661ce05bf5","Type":"ContainerStarted","Data":"608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe"} Jul 21 21:38:44.751730 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:44.751378 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" event={"ID":"65409ab7-338e-444e-8ba3-16661ce05bf5","Type":"ContainerStarted","Data":"c0cb5f5360a9f28e862bb264fa0bf458e9b3e3fc7795e21ff085a9d43ee5f9fa"} Jul 21 21:38:44.751730 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:44.751407 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:44.774960 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:44.774916 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" podStartSLOduration=2.438141226 podStartE2EDuration="2.774900822s" totalImagesPullingTime="336.759596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:38:44.774160071 +0000 UTC m=+2301.718781155" watchObservedRunningTime="2026-07-21 21:38:44.774900822 +0000 UTC m=+2301.719521886" Jul 21 21:38:47.120001 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:47.119969 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:47.931109 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:47.931072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:50.762947 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:50.762909 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:38:55.173245 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.173208 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz"] Jul 21 21:38:55.178452 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.178432 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:55.181484 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.181463 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:38:55.184734 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.184711 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz"] Jul 21 21:38:55.248431 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.248404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-226c3e-2trkz\" (UID: \"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:55.349298 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.349257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-226c3e-2trkz\" (UID: \"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:55.349962 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.349942 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-226c3e-2trkz\" (UID: \"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:55.492390 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.492274 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:55.629607 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.629578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz"] Jul 21 21:38:55.634053 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:38:55.633936 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebc30dc_1e9b_48ce_81bb_ef7a99e8923c.slice/crio-d78e8561fa880ec885765d9dc24488fed4051f6d2265a4941e0d5c8fb8c0c34a WatchSource:0}: Error finding container d78e8561fa880ec885765d9dc24488fed4051f6d2265a4941e0d5c8fb8c0c34a: Status 404 returned error can't find the container with id d78e8561fa880ec885765d9dc24488fed4051f6d2265a4941e0d5c8fb8c0c34a Jul 21 21:38:55.824710 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.824679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:55.827542 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.827508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" event={"ID":"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c","Type":"ContainerStarted","Data":"d78e8561fa880ec885765d9dc24488fed4051f6d2265a4941e0d5c8fb8c0c34a"} Jul 21 21:38:55.927170 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:55.927140 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:38:56.834226 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:56.834184 2570 generic.go:356] "Generic (PLEG): container finished" podID="9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c" containerID="4d7d2ea4115a62e7b9ed77f394b129b523d25d6895a99d7e4831e36825a84754" exitCode=0 Jul 21 21:38:56.834679 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:56.834273 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" event={"ID":"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c","Type":"ContainerDied","Data":"4d7d2ea4115a62e7b9ed77f394b129b523d25d6895a99d7e4831e36825a84754"} Jul 21 21:38:57.978320 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:57.978294 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:58.075436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:58.075403 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c-maas-api-service-ca\") pod \"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c\" (UID: \"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c\") " Jul 21 21:38:58.075771 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:58.075746 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c" (UID: "9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:38:58.176277 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:58.176242 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c-maas-api-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:38:58.846818 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:58.846790 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" Jul 21 21:38:58.846818 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:58.846816 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz" event={"ID":"9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c","Type":"ContainerDied","Data":"d78e8561fa880ec885765d9dc24488fed4051f6d2265a4941e0d5c8fb8c0c34a"} Jul 21 21:38:58.847090 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:38:58.846858 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d78e8561fa880ec885765d9dc24488fed4051f6d2265a4941e0d5c8fb8c0c34a" Jul 21 21:39:04.198102 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:04.198066 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:04.299668 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:04.299633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:05.716986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:05.716953 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:05.818919 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:05.818885 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:10.963549 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:10.963506 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7"] Jul 21 21:39:10.964003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:10.963724 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" podUID="10f78b72-a6de-4eba-a1e3-b228db477b40" containerName="maas-api" containerID="cri-o://4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0" gracePeriod=30 Jul 21 21:39:10.992307 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:10.992267 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:11.225955 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.225898 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:39:11.405149 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.405115 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f78b72-a6de-4eba-a1e3-b228db477b40-openshift-service-ca\") pod \"10f78b72-a6de-4eba-a1e3-b228db477b40\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " Jul 21 21:39:11.405334 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.405181 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbvwm\" (UniqueName: \"kubernetes.io/projected/10f78b72-a6de-4eba-a1e3-b228db477b40-kube-api-access-hbvwm\") pod \"10f78b72-a6de-4eba-a1e3-b228db477b40\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " Jul 21 21:39:11.405334 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.405221 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls\") pod \"10f78b72-a6de-4eba-a1e3-b228db477b40\" (UID: \"10f78b72-a6de-4eba-a1e3-b228db477b40\") " Jul 21 21:39:11.405518 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.405492 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f78b72-a6de-4eba-a1e3-b228db477b40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10f78b72-a6de-4eba-a1e3-b228db477b40" (UID: "10f78b72-a6de-4eba-a1e3-b228db477b40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:39:11.407272 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.407249 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f78b72-a6de-4eba-a1e3-b228db477b40-kube-api-access-hbvwm" (OuterVolumeSpecName: "kube-api-access-hbvwm") pod "10f78b72-a6de-4eba-a1e3-b228db477b40" (UID: "10f78b72-a6de-4eba-a1e3-b228db477b40"). InnerVolumeSpecName "kube-api-access-hbvwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:39:11.407409 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.407384 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10f78b72-a6de-4eba-a1e3-b228db477b40" (UID: "10f78b72-a6de-4eba-a1e3-b228db477b40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:39:11.506649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.506567 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f78b72-a6de-4eba-a1e3-b228db477b40-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:39:11.506649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.506594 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvwm\" (UniqueName: \"kubernetes.io/projected/10f78b72-a6de-4eba-a1e3-b228db477b40-kube-api-access-hbvwm\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:39:11.506649 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.506605 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10f78b72-a6de-4eba-a1e3-b228db477b40-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:39:11.921283 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.921252 2570 generic.go:356] "Generic (PLEG): container finished" podID="10f78b72-a6de-4eba-a1e3-b228db477b40" containerID="4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0" exitCode=0 Jul 21 21:39:11.921460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.921310 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" Jul 21 21:39:11.921460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.921342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" event={"ID":"10f78b72-a6de-4eba-a1e3-b228db477b40","Type":"ContainerDied","Data":"4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0"} Jul 21 21:39:11.921460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.921409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7" event={"ID":"10f78b72-a6de-4eba-a1e3-b228db477b40","Type":"ContainerDied","Data":"8a073379a784547953ac733ba785e96edcb83cec11e73c73e5acf57c40f3e1ab"} Jul 21 21:39:11.921460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.921433 2570 scope.go:117] "RemoveContainer" containerID="4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0" Jul 21 21:39:11.930268 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.930251 2570 scope.go:117] "RemoveContainer" containerID="4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0" Jul 21 21:39:11.930532 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:39:11.930514 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0\": container with ID starting with 4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0 not found: ID does not exist" containerID="4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0" Jul 21 21:39:11.930606 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.930539 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0"} err="failed to get container status \"4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0\": rpc error: code = NotFound desc = could not find container \"4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0\": container with ID starting with 4c1912b5619e7ef9ff38378d57348ee45b4bc694efe4d0bc79450ab85f288ae0 not found: ID does not exist" Jul 21 21:39:11.938952 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.938930 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7"] Jul 21 21:39:11.943560 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:11.943536 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-226c3e-784fd9c6f-k6wg7"] Jul 21 21:39:13.557589 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:13.557554 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f78b72-a6de-4eba-a1e3-b228db477b40" path="/var/lib/kubelet/pods/10f78b72-a6de-4eba-a1e3-b228db477b40/volumes" Jul 21 21:39:15.264964 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:15.264871 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz"] Jul 21 21:39:15.265506 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:15.264919 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:15.267932 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:15.267908 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-226c3e-2trkz"] Jul 21 21:39:15.557207 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:15.557130 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c" path="/var/lib/kubelet/pods/9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c/volumes" Jul 21 21:39:15.675290 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:15.675258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:21.547599 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:21.547557 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:21.647936 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:21.647899 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:21.866171 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:21.866089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:22.107894 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:22.107863 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:23.130525 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:23.130492 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:23.226923 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:23.226889 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:37.671518 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:37.671481 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl"] Jul 21 21:39:37.672093 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:37.671758 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" podUID="65409ab7-338e-444e-8ba3-16661ce05bf5" containerName="maas-api" containerID="cri-o://608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe" gracePeriod=30 Jul 21 21:39:37.698847 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:37.698821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:37.926451 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:37.926352 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:39:38.029171 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.029141 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65409ab7-338e-444e-8ba3-16661ce05bf5-openshift-service-ca\") pod \"65409ab7-338e-444e-8ba3-16661ce05bf5\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " Jul 21 21:39:38.029332 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.029201 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq9v2\" (UniqueName: \"kubernetes.io/projected/65409ab7-338e-444e-8ba3-16661ce05bf5-kube-api-access-wq9v2\") pod \"65409ab7-338e-444e-8ba3-16661ce05bf5\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " Jul 21 21:39:38.029332 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.029245 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls\") pod \"65409ab7-338e-444e-8ba3-16661ce05bf5\" (UID: \"65409ab7-338e-444e-8ba3-16661ce05bf5\") " Jul 21 21:39:38.029510 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.029486 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65409ab7-338e-444e-8ba3-16661ce05bf5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65409ab7-338e-444e-8ba3-16661ce05bf5" (UID: "65409ab7-338e-444e-8ba3-16661ce05bf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:39:38.031609 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.031585 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65409ab7-338e-444e-8ba3-16661ce05bf5" (UID: "65409ab7-338e-444e-8ba3-16661ce05bf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:39:38.031609 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.031592 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65409ab7-338e-444e-8ba3-16661ce05bf5-kube-api-access-wq9v2" (OuterVolumeSpecName: "kube-api-access-wq9v2") pod "65409ab7-338e-444e-8ba3-16661ce05bf5" (UID: "65409ab7-338e-444e-8ba3-16661ce05bf5"). InnerVolumeSpecName "kube-api-access-wq9v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:39:38.061661 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.061630 2570 generic.go:356] "Generic (PLEG): container finished" podID="65409ab7-338e-444e-8ba3-16661ce05bf5" containerID="608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe" exitCode=0 Jul 21 21:39:38.061754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.061675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" event={"ID":"65409ab7-338e-444e-8ba3-16661ce05bf5","Type":"ContainerDied","Data":"608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe"} Jul 21 21:39:38.061754 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.061682 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" Jul 21 21:39:38.061829 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.061761 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl" event={"ID":"65409ab7-338e-444e-8ba3-16661ce05bf5","Type":"ContainerDied","Data":"c0cb5f5360a9f28e862bb264fa0bf458e9b3e3fc7795e21ff085a9d43ee5f9fa"} Jul 21 21:39:38.061829 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.061791 2570 scope.go:117] "RemoveContainer" containerID="608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe" Jul 21 21:39:38.070902 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.070886 2570 scope.go:117] "RemoveContainer" containerID="608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe" Jul 21 21:39:38.071172 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:39:38.071152 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe\": container with ID starting with 608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe not found: ID does not exist" containerID="608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe" Jul 21 21:39:38.071236 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.071183 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe"} err="failed to get container status \"608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe\": rpc error: code = NotFound desc = could not find container \"608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe\": container with ID starting with 608a77d16f34db09c0ede7ff948ea03d540f8510506bab3bea862deae5e3fbfe not found: ID does not exist" Jul 21 21:39:38.085133 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.085102 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl"] Jul 21 21:39:38.089201 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.089181 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5898c4-7bfbc99f98-82rfl"] Jul 21 21:39:38.130708 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.130685 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65409ab7-338e-444e-8ba3-16661ce05bf5-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:39:38.130708 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.130707 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9v2\" (UniqueName: \"kubernetes.io/projected/65409ab7-338e-444e-8ba3-16661ce05bf5-kube-api-access-wq9v2\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:39:38.130824 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:38.130718 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65409ab7-338e-444e-8ba3-16661ce05bf5-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:39:39.556499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:39.556468 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65409ab7-338e-444e-8ba3-16661ce05bf5" path="/var/lib/kubelet/pods/65409ab7-338e-444e-8ba3-16661ce05bf5/volumes" Jul 21 21:39:42.306995 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:42.306961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:42.720927 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:42.720890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:48.509053 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:48.509016 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:48.608763 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:48.608724 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:48.792744 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:48.792669 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:49.044118 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:49.044032 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:49.393029 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:49.392937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:49.737781 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:49.737745 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:50.007803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:50.007725 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:52.984478 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:52.984443 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:53.088300 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.088264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:53.158175 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158145 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs"] Jul 21 21:39:53.158656 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158637 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f78b72-a6de-4eba-a1e3-b228db477b40" containerName="maas-api" Jul 21 21:39:53.158749 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158660 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f78b72-a6de-4eba-a1e3-b228db477b40" containerName="maas-api" Jul 21 21:39:53.158749 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158682 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65409ab7-338e-444e-8ba3-16661ce05bf5" containerName="maas-api" Jul 21 21:39:53.158749 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158692 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="65409ab7-338e-444e-8ba3-16661ce05bf5" containerName="maas-api" Jul 21 21:39:53.158749 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158718 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c" containerName="revoke-keys" Jul 21 21:39:53.158749 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158728 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c" containerName="revoke-keys" Jul 21 21:39:53.159021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158796 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="65409ab7-338e-444e-8ba3-16661ce05bf5" containerName="maas-api" Jul 21 21:39:53.159021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158805 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebc30dc-1e9b-48ce-81bb-ef7a99e8923c" containerName="revoke-keys" Jul 21 21:39:53.159021 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.158814 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f78b72-a6de-4eba-a1e3-b228db477b40" containerName="maas-api" Jul 21 21:39:53.163271 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.163251 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.166145 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.166125 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e584ac-serving-cert\"" Jul 21 21:39:53.174626 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.174602 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs"] Jul 21 21:39:53.198094 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.198063 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:53.259197 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.259111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.259434 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.259412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20897192-d86c-4434-87ce-a7535289f580-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.259527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.259440 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6lv\" (UniqueName: \"kubernetes.io/projected/20897192-d86c-4434-87ce-a7535289f580-kube-api-access-zk6lv\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.360831 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.360794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.360987 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.360851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20897192-d86c-4434-87ce-a7535289f580-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.360987 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:39:53.360953 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-serving-cert: secret "maas-api-e2e-shared-a-e584ac-serving-cert" not found Jul 21 21:39:53.361064 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:39:53.361022 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls podName:20897192-d86c-4434-87ce-a7535289f580 nodeName:}" failed. No retries permitted until 2026-07-21 21:39:53.861004836 +0000 UTC m=+2370.805625878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls") pod "maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" (UID: "20897192-d86c-4434-87ce-a7535289f580") : secret "maas-api-e2e-shared-a-e584ac-serving-cert" not found Jul 21 21:39:53.361121 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.361092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6lv\" (UniqueName: \"kubernetes.io/projected/20897192-d86c-4434-87ce-a7535289f580-kube-api-access-zk6lv\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.361499 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.361480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20897192-d86c-4434-87ce-a7535289f580-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.372321 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.372296 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6lv\" (UniqueName: \"kubernetes.io/projected/20897192-d86c-4434-87ce-a7535289f580-kube-api-access-zk6lv\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.682572 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.682546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:53.807510 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.807477 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:53.865241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.865213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.867719 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.867691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls\") pod \"maas-api-e2e-shared-a-e584ac-f67687549-ghjbs\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:53.908302 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:53.908274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:54.074570 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:54.074488 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:54.217696 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:54.217667 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs"] Jul 21 21:39:54.219213 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:39:54.219186 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20897192_d86c_4434_87ce_a7535289f580.slice/crio-ea2f5a49da8cd087fa9e254172f12a2f4794f0ffdf2da56bd57beee8f41fbcab WatchSource:0}: Error finding container ea2f5a49da8cd087fa9e254172f12a2f4794f0ffdf2da56bd57beee8f41fbcab: Status 404 returned error can't find the container with id ea2f5a49da8cd087fa9e254172f12a2f4794f0ffdf2da56bd57beee8f41fbcab Jul 21 21:39:55.162981 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:55.162947 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" event={"ID":"20897192-d86c-4434-87ce-a7535289f580","Type":"ContainerStarted","Data":"6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c"} Jul 21 21:39:55.163409 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:55.162997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" event={"ID":"20897192-d86c-4434-87ce-a7535289f580","Type":"ContainerStarted","Data":"ea2f5a49da8cd087fa9e254172f12a2f4794f0ffdf2da56bd57beee8f41fbcab"} Jul 21 21:39:55.163409 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:55.163052 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:39:55.181121 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:55.181073 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" podStartSLOduration=1.750809874 podStartE2EDuration="2.181059224s" totalImagesPullingTime="430.24935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:39:55.179647334 +0000 UTC m=+2372.124268396" watchObservedRunningTime="2026-07-21 21:39:55.181059224 +0000 UTC m=+2372.125680291" Jul 21 21:39:57.528508 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:57.528465 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:57.856906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:57.856817 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:58.185157 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:58.185123 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:39:58.620231 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:39:58.620140 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:01.055006 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.054971 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:01.157871 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.157837 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:01.175395 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.175345 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:40:01.253050 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.253019 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7"] Jul 21 21:40:01.256979 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.256954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.259579 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.259557 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-24ec74-serving-cert\"" Jul 21 21:40:01.269947 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.269920 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7"] Jul 21 21:40:01.295939 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.295912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:01.337098 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.337021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11abd2be-ab0f-4f25-848f-c5f459c52e4b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.337098 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.337073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2td\" (UniqueName: \"kubernetes.io/projected/11abd2be-ab0f-4f25-848f-c5f459c52e4b-kube-api-access-4g2td\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.337290 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.337114 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.438591 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.438554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11abd2be-ab0f-4f25-848f-c5f459c52e4b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.438591 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.438590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2td\" (UniqueName: \"kubernetes.io/projected/11abd2be-ab0f-4f25-848f-c5f459c52e4b-kube-api-access-4g2td\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.438815 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.438616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.438815 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:40:01.438709 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-serving-cert: secret "maas-api-e2e-shared-b-24ec74-serving-cert" not found Jul 21 21:40:01.438815 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:40:01.438762 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls podName:11abd2be-ab0f-4f25-848f-c5f459c52e4b nodeName:}" failed. No retries permitted until 2026-07-21 21:40:01.938746616 +0000 UTC m=+2378.883367658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls") pod "maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" (UID: "11abd2be-ab0f-4f25-848f-c5f459c52e4b") : secret "maas-api-e2e-shared-b-24ec74-serving-cert" not found Jul 21 21:40:01.439251 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.439230 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11abd2be-ab0f-4f25-848f-c5f459c52e4b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.468417 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.468384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2td\" (UniqueName: \"kubernetes.io/projected/11abd2be-ab0f-4f25-848f-c5f459c52e4b-kube-api-access-4g2td\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.833294 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.833256 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:01.933600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.933548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:01.943815 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.943789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:01.946432 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:01.946415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:02.168553 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:02.168513 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:02.313645 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:02.313617 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7"] Jul 21 21:40:03.211054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:03.211011 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" event={"ID":"11abd2be-ab0f-4f25-848f-c5f459c52e4b","Type":"ContainerStarted","Data":"ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b"} Jul 21 21:40:03.211538 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:03.211069 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" event={"ID":"11abd2be-ab0f-4f25-848f-c5f459c52e4b","Type":"ContainerStarted","Data":"24238814e5fbf10116dc10d9025db4a3f34a8c1c2123a2bc4bce8bfd0432dbdf"} Jul 21 21:40:03.211538 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:03.211167 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:03.231997 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:03.231947 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" podStartSLOduration=1.439850453 podStartE2EDuration="2.231933487s" totalImagesPullingTime="792.083034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:40:03.230204774 +0000 UTC m=+2380.174825839" watchObservedRunningTime="2026-07-21 21:40:03.231933487 +0000 UTC m=+2380.176554551" Jul 21 21:40:05.553601 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:05.553565 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:09.223313 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:09.223278 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:40:14.588235 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:14.588198 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:14.689111 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:14.689075 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:22.701125 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:22.701086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:22.804320 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:22.804276 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:23.572411 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:23.572349 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:23.673373 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:23.673324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:24.347991 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:24.347958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:24.449237 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:24.449198 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:30.763150 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:30.763116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:40:53.193095 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:40:53.193053 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:42:03.447105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:42:03.447065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:04.355497 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:04.355463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:05.607967 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:05.607923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:07.840349 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:07.840316 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:07.905504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:07.905471 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:15.142764 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:15.142727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:17.751062 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:17.751025 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:24.132250 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:24.132214 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:43:31.657148 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:43:31.657099 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:44:09.112831 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:44:09.112794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:44:10.151190 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:44:10.151159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:44:19.359853 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:44:19.359813 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:44:23.862300 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:44:23.862265 2570 scope.go:117] "RemoveContainer" containerID="c6b515a90762f11b3daae7732382bfa3b406da5eaa8bca30e43ee423dc5b2250" Jul 21 21:44:42.465198 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:44:42.465156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:44:44.727382 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:44:44.727311 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:00.133393 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.133333 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj"] Jul 21 21:45:00.136912 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.136894 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq"] Jul 21 21:45:00.137024 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.137002 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" Jul 21 21:45:00.139835 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.139817 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:45:00.140066 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.140044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" Jul 21 21:45:00.144672 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.144647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj"] Jul 21 21:45:00.149416 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.149391 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq"] Jul 21 21:45:00.150849 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.150830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" Jul 21 21:45:00.159941 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.159919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" Jul 21 21:45:00.311532 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.310764 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj"] Jul 21 21:45:00.313883 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:45:00.313850 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbe79bd_b879_46f7_b48e_e1b35f550892.slice/crio-92f853056239630475b4d4647bddefe35af21c9a2dcd0802423c3ec21adfa6c5 WatchSource:0}: Error finding container 92f853056239630475b4d4647bddefe35af21c9a2dcd0802423c3ec21adfa6c5: Status 404 returned error can't find the container with id 92f853056239630475b4d4647bddefe35af21c9a2dcd0802423c3ec21adfa6c5 Jul 21 21:45:00.547048 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.547025 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq"] Jul 21 21:45:00.548662 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:45:00.548631 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f894c82_13a0_4f6e_b08e_2b40bc787400.slice/crio-6f9d1b2311cc9631198c3ef682cec7abdeb455184c34b900cabc1967ded56d2b WatchSource:0}: Error finding container 6f9d1b2311cc9631198c3ef682cec7abdeb455184c34b900cabc1967ded56d2b: Status 404 returned error can't find the container with id 6f9d1b2311cc9631198c3ef682cec7abdeb455184c34b900cabc1967ded56d2b Jul 21 21:45:00.910210 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.910174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" event={"ID":"7fbe79bd-b879-46f7-b48e-e1b35f550892","Type":"ContainerStarted","Data":"b55d5847864981aa2d60f5e3ddbdb221dd1eab1b331e5050a12629bad0a30659"} Jul 21 21:45:00.910352 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.910241 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" event={"ID":"7fbe79bd-b879-46f7-b48e-e1b35f550892","Type":"ContainerStarted","Data":"92f853056239630475b4d4647bddefe35af21c9a2dcd0802423c3ec21adfa6c5"} Jul 21 21:45:00.911712 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.911689 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" event={"ID":"6f894c82-13a0-4f6e-b08e-2b40bc787400","Type":"ContainerStarted","Data":"b7594137c7883a6dff46a9f3f19448570f89fafb4b54f2b4f7cafbcc66cafcff"} Jul 21 21:45:00.911832 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.911730 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" event={"ID":"6f894c82-13a0-4f6e-b08e-2b40bc787400","Type":"ContainerStarted","Data":"6f9d1b2311cc9631198c3ef682cec7abdeb455184c34b900cabc1967ded56d2b"} Jul 21 21:45:00.927096 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.927048 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" podStartSLOduration=0.927031784 podStartE2EDuration="927.031784ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:45:00.925039438 +0000 UTC m=+2677.869660502" watchObservedRunningTime="2026-07-21 21:45:00.927031784 +0000 UTC m=+2677.871652851" Jul 21 21:45:00.940332 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:00.940287 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" podStartSLOduration=0.940274243 podStartE2EDuration="940.274243ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:45:00.939522651 +0000 UTC m=+2677.884143720" watchObservedRunningTime="2026-07-21 21:45:00.940274243 +0000 UTC m=+2677.884895308" Jul 21 21:45:01.918496 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:01.918456 2570 generic.go:356] "Generic (PLEG): container finished" podID="7fbe79bd-b879-46f7-b48e-e1b35f550892" containerID="b55d5847864981aa2d60f5e3ddbdb221dd1eab1b331e5050a12629bad0a30659" exitCode=0 Jul 21 21:45:01.918910 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:01.918550 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" event={"ID":"7fbe79bd-b879-46f7-b48e-e1b35f550892","Type":"ContainerDied","Data":"b55d5847864981aa2d60f5e3ddbdb221dd1eab1b331e5050a12629bad0a30659"} Jul 21 21:45:01.920071 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:01.920049 2570 generic.go:356] "Generic (PLEG): container finished" podID="6f894c82-13a0-4f6e-b08e-2b40bc787400" containerID="b7594137c7883a6dff46a9f3f19448570f89fafb4b54f2b4f7cafbcc66cafcff" exitCode=0 Jul 21 21:45:01.920163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:01.920129 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" event={"ID":"6f894c82-13a0-4f6e-b08e-2b40bc787400","Type":"ContainerDied","Data":"b7594137c7883a6dff46a9f3f19448570f89fafb4b54f2b4f7cafbcc66cafcff"} Jul 21 21:45:03.074473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.074443 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" Jul 21 21:45:03.095571 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.095543 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" Jul 21 21:45:03.933036 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.933005 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" Jul 21 21:45:03.933224 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.933050 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744505-cd2jq" event={"ID":"6f894c82-13a0-4f6e-b08e-2b40bc787400","Type":"ContainerDied","Data":"6f9d1b2311cc9631198c3ef682cec7abdeb455184c34b900cabc1967ded56d2b"} Jul 21 21:45:03.933224 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.933104 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f9d1b2311cc9631198c3ef682cec7abdeb455184c34b900cabc1967ded56d2b" Jul 21 21:45:03.934436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.934414 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" event={"ID":"7fbe79bd-b879-46f7-b48e-e1b35f550892","Type":"ContainerDied","Data":"92f853056239630475b4d4647bddefe35af21c9a2dcd0802423c3ec21adfa6c5"} Jul 21 21:45:03.934436 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.934435 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj" Jul 21 21:45:03.934633 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:03.934451 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f853056239630475b4d4647bddefe35af21c9a2dcd0802423c3ec21adfa6c5" Jul 21 21:45:15.356062 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:15.355972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:15.456324 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:15.456288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:23.508958 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:23.508917 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:23.609203 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:23.609165 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:23.886594 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:23.886476 2570 scope.go:117] "RemoveContainer" containerID="4d7d2ea4115a62e7b9ed77f394b129b523d25d6895a99d7e4831e36825a84754" Jul 21 21:45:24.439149 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:24.439109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:24.539197 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:24.539161 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:29.531861 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:29.531825 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:30.365504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:30.365469 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:50.045666 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.045631 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl"] Jul 21 21:45:50.046258 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.046237 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbe79bd-b879-46f7-b48e-e1b35f550892" containerName="cleanup" Jul 21 21:45:50.046311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.046263 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbe79bd-b879-46f7-b48e-e1b35f550892" containerName="cleanup" Jul 21 21:45:50.046311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.046277 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f894c82-13a0-4f6e-b08e-2b40bc787400" containerName="cleanup" Jul 21 21:45:50.046311 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.046285 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f894c82-13a0-4f6e-b08e-2b40bc787400" containerName="cleanup" Jul 21 21:45:50.046451 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.046410 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbe79bd-b879-46f7-b48e-e1b35f550892" containerName="cleanup" Jul 21 21:45:50.046451 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.046423 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f894c82-13a0-4f6e-b08e-2b40bc787400" containerName="cleanup" Jul 21 21:45:50.049732 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.049713 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:50.052572 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.052551 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:45:50.058298 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.058274 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl"] Jul 21 21:45:50.081481 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.081442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a2fbf3-90f9-4766-9b64-b5e23d9f0704-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl\" (UID: \"65a2fbf3-90f9-4766-9b64-b5e23d9f0704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:50.181978 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.181938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a2fbf3-90f9-4766-9b64-b5e23d9f0704-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl\" (UID: \"65a2fbf3-90f9-4766-9b64-b5e23d9f0704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:50.182624 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.182606 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a2fbf3-90f9-4766-9b64-b5e23d9f0704-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl\" (UID: \"65a2fbf3-90f9-4766-9b64-b5e23d9f0704\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:50.361984 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.361894 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:50.503221 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.503183 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl"] Jul 21 21:45:50.504412 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:45:50.504382 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a2fbf3_90f9_4766_9b64_b5e23d9f0704.slice/crio-1bf1abd008b2b9c7d9250b469ded057965ddb2133b8d6181f0a32c4614d7e59c WatchSource:0}: Error finding container 1bf1abd008b2b9c7d9250b469ded057965ddb2133b8d6181f0a32c4614d7e59c: Status 404 returned error can't find the container with id 1bf1abd008b2b9c7d9250b469ded057965ddb2133b8d6181f0a32c4614d7e59c Jul 21 21:45:50.718163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.718128 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:50.821833 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:50.821804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:45:51.197308 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:51.197273 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" event={"ID":"65a2fbf3-90f9-4766-9b64-b5e23d9f0704","Type":"ContainerStarted","Data":"28ae668b202ac8d458c5e00a40323ac93583cc85a7f5dbe46fc0ed1eae5a8767"} Jul 21 21:45:51.197730 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:51.197343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" event={"ID":"65a2fbf3-90f9-4766-9b64-b5e23d9f0704","Type":"ContainerStarted","Data":"1bf1abd008b2b9c7d9250b469ded057965ddb2133b8d6181f0a32c4614d7e59c"} Jul 21 21:45:51.217823 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:51.217780 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" podStartSLOduration=1.217765972 podStartE2EDuration="1.217765972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:45:51.21617588 +0000 UTC m=+2728.160796946" watchObservedRunningTime="2026-07-21 21:45:51.217765972 +0000 UTC m=+2728.162387036" Jul 21 21:45:52.204581 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:52.204545 2570 generic.go:356] "Generic (PLEG): container finished" podID="65a2fbf3-90f9-4766-9b64-b5e23d9f0704" containerID="28ae668b202ac8d458c5e00a40323ac93583cc85a7f5dbe46fc0ed1eae5a8767" exitCode=0 Jul 21 21:45:52.205008 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:52.204644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" event={"ID":"65a2fbf3-90f9-4766-9b64-b5e23d9f0704","Type":"ContainerDied","Data":"28ae668b202ac8d458c5e00a40323ac93583cc85a7f5dbe46fc0ed1eae5a8767"} Jul 21 21:45:53.348986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:53.348959 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:53.409783 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:53.409750 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a2fbf3-90f9-4766-9b64-b5e23d9f0704-maas-api-service-ca\") pod \"65a2fbf3-90f9-4766-9b64-b5e23d9f0704\" (UID: \"65a2fbf3-90f9-4766-9b64-b5e23d9f0704\") " Jul 21 21:45:53.410108 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:53.410082 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a2fbf3-90f9-4766-9b64-b5e23d9f0704-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65a2fbf3-90f9-4766-9b64-b5e23d9f0704" (UID: "65a2fbf3-90f9-4766-9b64-b5e23d9f0704"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:45:53.511193 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:53.511107 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a2fbf3-90f9-4766-9b64-b5e23d9f0704-maas-api-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:45:54.218301 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:54.218273 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" Jul 21 21:45:54.218534 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:54.218264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl" event={"ID":"65a2fbf3-90f9-4766-9b64-b5e23d9f0704","Type":"ContainerDied","Data":"1bf1abd008b2b9c7d9250b469ded057965ddb2133b8d6181f0a32c4614d7e59c"} Jul 21 21:45:54.218534 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:45:54.218416 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bf1abd008b2b9c7d9250b469ded057965ddb2133b8d6181f0a32c4614d7e59c" Jul 21 21:46:00.495092 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.495039 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs"] Jul 21 21:46:00.495599 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.495403 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" podUID="20897192-d86c-4434-87ce-a7535289f580" containerName="maas-api" containerID="cri-o://6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c" gracePeriod=30 Jul 21 21:46:00.526858 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.526822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:00.755567 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.755500 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:46:00.879460 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.879425 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20897192-d86c-4434-87ce-a7535289f580-openshift-service-ca\") pod \"20897192-d86c-4434-87ce-a7535289f580\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " Jul 21 21:46:00.879657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.879498 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls\") pod \"20897192-d86c-4434-87ce-a7535289f580\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " Jul 21 21:46:00.879657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.879522 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk6lv\" (UniqueName: \"kubernetes.io/projected/20897192-d86c-4434-87ce-a7535289f580-kube-api-access-zk6lv\") pod \"20897192-d86c-4434-87ce-a7535289f580\" (UID: \"20897192-d86c-4434-87ce-a7535289f580\") " Jul 21 21:46:00.879836 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.879808 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20897192-d86c-4434-87ce-a7535289f580-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20897192-d86c-4434-87ce-a7535289f580" (UID: "20897192-d86c-4434-87ce-a7535289f580"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:46:00.881694 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.881670 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20897192-d86c-4434-87ce-a7535289f580-kube-api-access-zk6lv" (OuterVolumeSpecName: "kube-api-access-zk6lv") pod "20897192-d86c-4434-87ce-a7535289f580" (UID: "20897192-d86c-4434-87ce-a7535289f580"). InnerVolumeSpecName "kube-api-access-zk6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:46:00.881818 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.881695 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20897192-d86c-4434-87ce-a7535289f580" (UID: "20897192-d86c-4434-87ce-a7535289f580"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:46:00.981003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.980964 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20897192-d86c-4434-87ce-a7535289f580-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:00.981003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.980996 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk6lv\" (UniqueName: \"kubernetes.io/projected/20897192-d86c-4434-87ce-a7535289f580-kube-api-access-zk6lv\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:00.981003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:00.981005 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20897192-d86c-4434-87ce-a7535289f580-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:01.257613 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.257579 2570 generic.go:356] "Generic (PLEG): container finished" podID="20897192-d86c-4434-87ce-a7535289f580" containerID="6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c" exitCode=0 Jul 21 21:46:01.257897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.257629 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" Jul 21 21:46:01.257897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.257662 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" event={"ID":"20897192-d86c-4434-87ce-a7535289f580","Type":"ContainerDied","Data":"6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c"} Jul 21 21:46:01.257897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.257718 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs" event={"ID":"20897192-d86c-4434-87ce-a7535289f580","Type":"ContainerDied","Data":"ea2f5a49da8cd087fa9e254172f12a2f4794f0ffdf2da56bd57beee8f41fbcab"} Jul 21 21:46:01.257897 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.257740 2570 scope.go:117] "RemoveContainer" containerID="6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c" Jul 21 21:46:01.267036 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.267016 2570 scope.go:117] "RemoveContainer" containerID="6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c" Jul 21 21:46:01.267299 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:46:01.267280 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c\": container with ID starting with 6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c not found: ID does not exist" containerID="6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c" Jul 21 21:46:01.267383 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.267311 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c"} err="failed to get container status \"6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c\": rpc error: code = NotFound desc = could not find container \"6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c\": container with ID starting with 6400524ed8fef0f86c24caac3cc6ad19cea09190789b53376e2ed544ed05801c not found: ID does not exist" Jul 21 21:46:01.281396 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.281350 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs"] Jul 21 21:46:01.284888 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.284862 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e584ac-f67687549-ghjbs"] Jul 21 21:46:01.557525 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:01.557447 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20897192-d86c-4434-87ce-a7535289f580" path="/var/lib/kubelet/pods/20897192-d86c-4434-87ce-a7535289f580/volumes" Jul 21 21:46:05.145500 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:05.145455 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:05.157924 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:05.157892 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl"] Jul 21 21:46:05.160140 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:05.160116 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e584ac-29gjl"] Jul 21 21:46:05.557593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:05.557558 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a2fbf3-90f9-4766-9b64-b5e23d9f0704" path="/var/lib/kubelet/pods/65a2fbf3-90f9-4766-9b64-b5e23d9f0704/volumes" Jul 21 21:46:05.653950 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:05.653917 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:11.706714 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:11.706681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:11.780948 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:11.780912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:12.006091 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.006013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:12.242051 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.242011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:12.379964 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.379877 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7"] Jul 21 21:46:12.380321 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.380307 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a2fbf3-90f9-4766-9b64-b5e23d9f0704" containerName="revoke-keys" Jul 21 21:46:12.380387 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.380323 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a2fbf3-90f9-4766-9b64-b5e23d9f0704" containerName="revoke-keys" Jul 21 21:46:12.380387 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.380333 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20897192-d86c-4434-87ce-a7535289f580" containerName="maas-api" Jul 21 21:46:12.380387 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.380340 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="20897192-d86c-4434-87ce-a7535289f580" containerName="maas-api" Jul 21 21:46:12.380483 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.380416 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a2fbf3-90f9-4766-9b64-b5e23d9f0704" containerName="revoke-keys" Jul 21 21:46:12.380483 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.380423 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="20897192-d86c-4434-87ce-a7535289f580" containerName="maas-api" Jul 21 21:46:12.385003 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.384982 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:12.387761 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.387741 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:46:12.390977 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.390956 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7"] Jul 21 21:46:12.491344 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.491308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1727ad86-3ce5-4239-9948-fe9f2d00ecd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7\" (UID: \"1727ad86-3ce5-4239-9948-fe9f2d00ecd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:12.592068 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.592025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1727ad86-3ce5-4239-9948-fe9f2d00ecd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7\" (UID: \"1727ad86-3ce5-4239-9948-fe9f2d00ecd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:12.592670 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.592651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1727ad86-3ce5-4239-9948-fe9f2d00ecd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7\" (UID: \"1727ad86-3ce5-4239-9948-fe9f2d00ecd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:12.697697 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.697652 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:12.838666 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:12.838636 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7"] Jul 21 21:46:12.839006 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:46:12.838781 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1727ad86_3ce5_4239_9948_fe9f2d00ecd5.slice/crio-bc5bad611f9d32ff3ea712cd6571f3c57018a552c8457d69f42fe49c1c6b6acd WatchSource:0}: Error finding container bc5bad611f9d32ff3ea712cd6571f3c57018a552c8457d69f42fe49c1c6b6acd: Status 404 returned error can't find the container with id bc5bad611f9d32ff3ea712cd6571f3c57018a552c8457d69f42fe49c1c6b6acd Jul 21 21:46:13.061254 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:13.061223 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:13.159859 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:13.159819 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:13.332695 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:13.332600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" event={"ID":"1727ad86-3ce5-4239-9948-fe9f2d00ecd5","Type":"ContainerStarted","Data":"bbde70de2aa9f059a27ff065947201b8e36c34299eb394037c1a950b44c117f4"} Jul 21 21:46:13.332695 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:13.332665 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" event={"ID":"1727ad86-3ce5-4239-9948-fe9f2d00ecd5","Type":"ContainerStarted","Data":"bc5bad611f9d32ff3ea712cd6571f3c57018a552c8457d69f42fe49c1c6b6acd"} Jul 21 21:46:13.348594 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:13.348552 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" podStartSLOduration=1.348537563 podStartE2EDuration="1.348537563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:46:13.347578671 +0000 UTC m=+2750.292199737" watchObservedRunningTime="2026-07-21 21:46:13.348537563 +0000 UTC m=+2750.293158627" Jul 21 21:46:14.339449 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:14.339417 2570 generic.go:356] "Generic (PLEG): container finished" podID="1727ad86-3ce5-4239-9948-fe9f2d00ecd5" containerID="bbde70de2aa9f059a27ff065947201b8e36c34299eb394037c1a950b44c117f4" exitCode=0 Jul 21 21:46:14.339860 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:14.339543 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" event={"ID":"1727ad86-3ce5-4239-9948-fe9f2d00ecd5","Type":"ContainerDied","Data":"bbde70de2aa9f059a27ff065947201b8e36c34299eb394037c1a950b44c117f4"} Jul 21 21:46:15.485843 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:15.485819 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:15.618416 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:15.618312 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1727ad86-3ce5-4239-9948-fe9f2d00ecd5-maas-api-service-ca\") pod \"1727ad86-3ce5-4239-9948-fe9f2d00ecd5\" (UID: \"1727ad86-3ce5-4239-9948-fe9f2d00ecd5\") " Jul 21 21:46:15.618672 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:15.618646 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1727ad86-3ce5-4239-9948-fe9f2d00ecd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1727ad86-3ce5-4239-9948-fe9f2d00ecd5" (UID: "1727ad86-3ce5-4239-9948-fe9f2d00ecd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:46:15.720073 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:15.720037 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1727ad86-3ce5-4239-9948-fe9f2d00ecd5-maas-api-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:16.352793 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:16.352761 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" Jul 21 21:46:16.352977 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:16.352800 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7" event={"ID":"1727ad86-3ce5-4239-9948-fe9f2d00ecd5","Type":"ContainerDied","Data":"bc5bad611f9d32ff3ea712cd6571f3c57018a552c8457d69f42fe49c1c6b6acd"} Jul 21 21:46:16.352977 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:16.352874 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5bad611f9d32ff3ea712cd6571f3c57018a552c8457d69f42fe49c1c6b6acd" Jul 21 21:46:22.840483 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:22.840450 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7"] Jul 21 21:46:22.841030 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:22.840728 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" podUID="11abd2be-ab0f-4f25-848f-c5f459c52e4b" containerName="maas-api" containerID="cri-o://ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b" gracePeriod=30 Jul 21 21:46:22.865900 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:22.865871 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj"] Jul 21 21:46:22.866738 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:22.866697 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:22.868421 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:22.868400 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-24ec74-29744505-rcwnj"] Jul 21 21:46:23.098895 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.098840 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:46:23.186429 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.186355 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11abd2be-ab0f-4f25-848f-c5f459c52e4b-openshift-service-ca\") pod \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " Jul 21 21:46:23.186606 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.186455 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g2td\" (UniqueName: \"kubernetes.io/projected/11abd2be-ab0f-4f25-848f-c5f459c52e4b-kube-api-access-4g2td\") pod \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " Jul 21 21:46:23.186606 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.186501 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls\") pod \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\" (UID: \"11abd2be-ab0f-4f25-848f-c5f459c52e4b\") " Jul 21 21:46:23.186767 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.186741 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11abd2be-ab0f-4f25-848f-c5f459c52e4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11abd2be-ab0f-4f25-848f-c5f459c52e4b" (UID: "11abd2be-ab0f-4f25-848f-c5f459c52e4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:46:23.186911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.186852 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11abd2be-ab0f-4f25-848f-c5f459c52e4b-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:23.188792 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.188746 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11abd2be-ab0f-4f25-848f-c5f459c52e4b" (UID: "11abd2be-ab0f-4f25-848f-c5f459c52e4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:46:23.188906 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.188785 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11abd2be-ab0f-4f25-848f-c5f459c52e4b-kube-api-access-4g2td" (OuterVolumeSpecName: "kube-api-access-4g2td") pod "11abd2be-ab0f-4f25-848f-c5f459c52e4b" (UID: "11abd2be-ab0f-4f25-848f-c5f459c52e4b"). InnerVolumeSpecName "kube-api-access-4g2td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:46:23.287279 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.287251 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2td\" (UniqueName: \"kubernetes.io/projected/11abd2be-ab0f-4f25-848f-c5f459c52e4b-kube-api-access-4g2td\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:23.287279 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.287276 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11abd2be-ab0f-4f25-848f-c5f459c52e4b-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 21 21:46:23.394527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.394434 2570 generic.go:356] "Generic (PLEG): container finished" podID="11abd2be-ab0f-4f25-848f-c5f459c52e4b" containerID="ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b" exitCode=0 Jul 21 21:46:23.394527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.394486 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" Jul 21 21:46:23.394711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.394523 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" event={"ID":"11abd2be-ab0f-4f25-848f-c5f459c52e4b","Type":"ContainerDied","Data":"ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b"} Jul 21 21:46:23.394711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.394585 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7" event={"ID":"11abd2be-ab0f-4f25-848f-c5f459c52e4b","Type":"ContainerDied","Data":"24238814e5fbf10116dc10d9025db4a3f34a8c1c2123a2bc4bce8bfd0432dbdf"} Jul 21 21:46:23.394711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.394611 2570 scope.go:117] "RemoveContainer" containerID="ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b" Jul 21 21:46:23.404902 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.404878 2570 scope.go:117] "RemoveContainer" containerID="ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b" Jul 21 21:46:23.405152 ip-10-0-132-65 kubenswrapper[2570]: E0721 21:46:23.405133 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b\": container with ID starting with ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b not found: ID does not exist" containerID="ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b" Jul 21 21:46:23.405209 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.405160 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b"} err="failed to get container status \"ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b\": rpc error: code = NotFound desc = could not find container \"ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b\": container with ID starting with ed7128120eeea0e58186262995dffc0b78b2d3bab03044424b0dc0be5b51f21b not found: ID does not exist" Jul 21 21:46:23.418987 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.418965 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7"] Jul 21 21:46:23.423790 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.423769 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-24ec74-64f7d5bfbf-fjjd7"] Jul 21 21:46:23.558520 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.558484 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11abd2be-ab0f-4f25-848f-c5f459c52e4b" path="/var/lib/kubelet/pods/11abd2be-ab0f-4f25-848f-c5f459c52e4b/volumes" Jul 21 21:46:23.558787 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.558773 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbe79bd-b879-46f7-b48e-e1b35f550892" path="/var/lib/kubelet/pods/7fbe79bd-b879-46f7-b48e-e1b35f550892/volumes" Jul 21 21:46:23.903921 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:23.903894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:27.462721 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:27.462664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:27.463163 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:27.462893 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7"] Jul 21 21:46:27.467657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:27.467633 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-24ec74-rthx7"] Jul 21 21:46:27.556727 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:27.556699 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1727ad86-3ce5-4239-9948-fe9f2d00ecd5" path="/var/lib/kubelet/pods/1727ad86-3ce5-4239-9948-fe9f2d00ecd5/volumes" Jul 21 21:46:27.892587 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:27.892480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:33.661186 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:33.661149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:33.735601 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:33.735574 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:33.952167 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:33.952107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:34.179666 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:34.179628 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:40.344458 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:40.344423 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pvvx7_bf1921af-bb51-43c6-b544-cecff337f092/manager/0.log" Jul 21 21:46:40.458759 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:40.458728 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r2v84_de3fe3cc-480a-44ab-bf75-c0a5d46e2118/manager/0.log" Jul 21 21:46:40.577538 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:40.577507 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55d6556f9b-bf9cf_5258ce45-e3fe-4709-a616-63a72bd46073/manager/0.log" Jul 21 21:46:40.686275 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:40.686250 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ss4kl_0d1239b8-a55a-462c-a1ab-8c0a241c6ac6/server/0.log" Jul 21 21:46:40.811017 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:40.810988 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fkrdd_516b1e47-edcc-42dd-a976-019883307b6b/manager/0.log" Jul 21 21:46:41.063886 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:41.063810 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-rrjnd_5258cfd5-86d0-4592-b8bc-2ea861d1a35d/copy-manifests/0.log" Jul 21 21:46:41.077847 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:41.077820 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-rrjnd_5258cfd5-86d0-4592-b8bc-2ea861d1a35d/manager/0.log" Jul 21 21:46:41.589520 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:41.589488 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8dff7b7df-kprrv_e494b6ca-a709-4c4d-af65-c026af2520e9/maas-api/0.log" Jul 21 21:46:41.939526 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:41.939494 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744505-cd2jq_6f894c82-13a0-4f6e-b08e-2b40bc787400/cleanup/0.log" Jul 21 21:46:42.064247 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:42.064220 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jpj6b_9d1351aa-977d-4ee9-a50b-ca7a5db6da56/postgres/0.log" Jul 21 21:46:42.900553 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:42.900520 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm_a0fbd1b0-6f1f-4da7-b882-78c89fc83c07/util/0.log" Jul 21 21:46:42.907994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:42.907966 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm_a0fbd1b0-6f1f-4da7-b882-78c89fc83c07/pull/0.log" Jul 21 21:46:42.914583 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:42.914565 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm_a0fbd1b0-6f1f-4da7-b882-78c89fc83c07/extract/0.log" Jul 21 21:46:43.033746 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.033720 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7_e9b00904-c24d-4ae8-ac42-1fe8995d8a32/util/0.log" Jul 21 21:46:43.040976 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.040954 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7_e9b00904-c24d-4ae8-ac42-1fe8995d8a32/pull/0.log" Jul 21 21:46:43.047662 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.047642 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7_e9b00904-c24d-4ae8-ac42-1fe8995d8a32/extract/0.log" Jul 21 21:46:43.174830 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.174800 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2_db02354b-33ea-41bc-9522-7623a1fdded8/util/0.log" Jul 21 21:46:43.182004 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.181984 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2_db02354b-33ea-41bc-9522-7623a1fdded8/pull/0.log" Jul 21 21:46:43.188600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.188555 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2_db02354b-33ea-41bc-9522-7623a1fdded8/extract/0.log" Jul 21 21:46:43.304755 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.304719 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x_3e950e96-708d-4bec-a54e-38b88d6d03f4/pull/0.log" Jul 21 21:46:43.311678 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.311657 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x_3e950e96-708d-4bec-a54e-38b88d6d03f4/extract/0.log" Jul 21 21:46:43.317759 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.317737 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x_3e950e96-708d-4bec-a54e-38b88d6d03f4/util/0.log" Jul 21 21:46:43.687045 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:43.687018 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-srz8n_2b43ab62-d073-495e-841e-f085c544f1c1/manager/0.log" Jul 21 21:46:44.334223 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:44.334192 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-55bt5_d7388aa8-9190-4187-82c6-a6b4d5005b50/manager/0.log" Jul 21 21:46:44.848732 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:44.848693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-clcpk_e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6/discovery/0.log" Jul 21 21:46:44.969009 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:44.968981 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b46444884-9dqzl_8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb/kube-auth-proxy/0.log" Jul 21 21:46:45.346326 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:45.346294 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hzfw4_54066cd0-970a-42c7-84a8-67bd17e03028/payload-pre-processing/0.log" Jul 21 21:46:45.463413 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:45.463385 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-g5x69_3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9/payload-processing/0.log" Jul 21 21:46:45.944284 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:45.944259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z4bpl_810765f3-e481-4551-8bcb-b850d3198e52/main/0.log" Jul 21 21:46:46.365911 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:46.365824 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2lqm8_b4b7293a-7545-4cfa-a9e3-74a10253a82d/main/0.log" Jul 21 21:46:52.904781 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:52.904741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:52.974865 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:52.974831 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" limit=1024 Jul 21 21:46:55.909623 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:55.909590 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6ddkj_77077877-720b-4a5b-a604-2c2b0af5ba0c/global-pull-secret-syncer/0.log" Jul 21 21:46:56.011600 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:56.011568 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8wghw_5ec019a3-7329-4860-a941-03bc6091f53f/konnectivity-agent/0.log" Jul 21 21:46:56.130684 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:46:56.130655 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-65.ec2.internal_5bb4be778fdc4d1b1c65a09ca6ed53e9/haproxy/0.log" Jul 21 21:47:00.081473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.081444 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm_a0fbd1b0-6f1f-4da7-b882-78c89fc83c07/extract/0.log" Jul 21 21:47:00.115488 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.115457 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm_a0fbd1b0-6f1f-4da7-b882-78c89fc83c07/util/0.log" Jul 21 21:47:00.138462 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.138435 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kchqm_a0fbd1b0-6f1f-4da7-b882-78c89fc83c07/pull/0.log" Jul 21 21:47:00.166002 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.165974 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7_e9b00904-c24d-4ae8-ac42-1fe8995d8a32/extract/0.log" Jul 21 21:47:00.191035 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.191009 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7_e9b00904-c24d-4ae8-ac42-1fe8995d8a32/util/0.log" Jul 21 21:47:00.212504 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.212477 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jnmh7_e9b00904-c24d-4ae8-ac42-1fe8995d8a32/pull/0.log" Jul 21 21:47:00.239105 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.239077 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2_db02354b-33ea-41bc-9522-7623a1fdded8/extract/0.log" Jul 21 21:47:00.262023 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.262000 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2_db02354b-33ea-41bc-9522-7623a1fdded8/util/0.log" Jul 21 21:47:00.286279 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.286255 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735zwc2_db02354b-33ea-41bc-9522-7623a1fdded8/pull/0.log" Jul 21 21:47:00.314751 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.314722 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x_3e950e96-708d-4bec-a54e-38b88d6d03f4/extract/0.log" Jul 21 21:47:00.338685 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.338616 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x_3e950e96-708d-4bec-a54e-38b88d6d03f4/util/0.log" Jul 21 21:47:00.363711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.363686 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qhk5x_3e950e96-708d-4bec-a54e-38b88d6d03f4/pull/0.log" Jul 21 21:47:00.785733 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:00.785703 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-srz8n_2b43ab62-d073-495e-841e-f085c544f1c1/manager/0.log" Jul 21 21:47:01.270063 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:01.270028 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-55bt5_d7388aa8-9190-4187-82c6-a6b4d5005b50/manager/0.log" Jul 21 21:47:03.383133 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.383059 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55f5b4445c-kp7vq_0160d1e1-ae4a-4a21-8b37-878ed30f763d/metrics-server/0.log" Jul 21 21:47:03.454241 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.454192 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78lvl_7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd/node-exporter/0.log" Jul 21 21:47:03.476515 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.476477 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78lvl_7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd/kube-rbac-proxy/0.log" Jul 21 21:47:03.497338 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.497315 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78lvl_7d33f92a-28a6-4959-ac85-1a2ec6d3b9bd/init-textfile/0.log" Jul 21 21:47:03.722173 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.722142 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ftzq4_ad54dbda-7a2f-44a8-9063-9d017a5617f9/kube-rbac-proxy-main/0.log" Jul 21 21:47:03.747738 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.747711 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ftzq4_ad54dbda-7a2f-44a8-9063-9d017a5617f9/kube-rbac-proxy-self/0.log" Jul 21 21:47:03.772589 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:03.772564 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ftzq4_ad54dbda-7a2f-44a8-9063-9d017a5617f9/openshift-state-metrics/0.log" Jul 21 21:47:04.214543 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:04.214516 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c9fd97-tdkpc_04b6e8fa-b025-4c01-9c71-c347a772a61f/thanos-query/0.log" Jul 21 21:47:04.239954 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:04.239926 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c9fd97-tdkpc_04b6e8fa-b025-4c01-9c71-c347a772a61f/kube-rbac-proxy-web/0.log" Jul 21 21:47:04.263146 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:04.263118 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c9fd97-tdkpc_04b6e8fa-b025-4c01-9c71-c347a772a61f/kube-rbac-proxy/0.log" Jul 21 21:47:04.289643 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:04.289616 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c9fd97-tdkpc_04b6e8fa-b025-4c01-9c71-c347a772a61f/prom-label-proxy/0.log" Jul 21 21:47:04.312464 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:04.312415 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c9fd97-tdkpc_04b6e8fa-b025-4c01-9c71-c347a772a61f/kube-rbac-proxy-rules/0.log" Jul 21 21:47:04.338979 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:04.338923 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c9fd97-tdkpc_04b6e8fa-b025-4c01-9c71-c347a772a61f/kube-rbac-proxy-metrics/0.log" Jul 21 21:47:06.207473 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.207429 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp"] Jul 21 21:47:06.208388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.208339 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1727ad86-3ce5-4239-9948-fe9f2d00ecd5" containerName="revoke-keys" Jul 21 21:47:06.208388 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.208382 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1727ad86-3ce5-4239-9948-fe9f2d00ecd5" containerName="revoke-keys" Jul 21 21:47:06.208565 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.208438 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11abd2be-ab0f-4f25-848f-c5f459c52e4b" containerName="maas-api" Jul 21 21:47:06.208565 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.208447 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="11abd2be-ab0f-4f25-848f-c5f459c52e4b" containerName="maas-api" Jul 21 21:47:06.208657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.208641 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="11abd2be-ab0f-4f25-848f-c5f459c52e4b" containerName="maas-api" Jul 21 21:47:06.208657 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.208655 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1727ad86-3ce5-4239-9948-fe9f2d00ecd5" containerName="revoke-keys" Jul 21 21:47:06.235091 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.235060 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.238587 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.238565 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4hjj\"/\"openshift-service-ca.crt\"" Jul 21 21:47:06.239288 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.239260 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp"] Jul 21 21:47:06.239849 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.239820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b4hjj\"/\"default-dockercfg-p8l7g\"" Jul 21 21:47:06.240054 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.240038 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4hjj\"/\"kube-root-ca.crt\"" Jul 21 21:47:06.373790 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.373748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-proc\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.374000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.373808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-sys\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.374000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.373858 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-podres\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.374000 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.373892 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2k5p\" (UniqueName: \"kubernetes.io/projected/fd4d768e-2159-4546-a387-a0db53507a10-kube-api-access-h2k5p\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.374148 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.373994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-lib-modules\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475336 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-lib-modules\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475336 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475307 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-proc\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-sys\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-proc\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475416 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-podres\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-sys\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475479 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-lib-modules\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2k5p\" (UniqueName: \"kubernetes.io/projected/fd4d768e-2159-4546-a387-a0db53507a10-kube-api-access-h2k5p\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.475604 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.475508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd4d768e-2159-4546-a387-a0db53507a10-podres\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.485588 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.485563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2k5p\" (UniqueName: \"kubernetes.io/projected/fd4d768e-2159-4546-a387-a0db53507a10-kube-api-access-h2k5p\") pod \"perf-node-gather-daemonset-g7mpp\" (UID: \"fd4d768e-2159-4546-a387-a0db53507a10\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.547527 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.547493 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:06.685439 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:06.685415 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp"] Jul 21 21:47:06.687492 ip-10-0-132-65 kubenswrapper[2570]: W0721 21:47:06.687450 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfd4d768e_2159_4546_a387_a0db53507a10.slice/crio-537ba7c4c7e207a7d02e8af0d9f562e1f8cfb4a76bd2ab5837fb5522ce90d945 WatchSource:0}: Error finding container 537ba7c4c7e207a7d02e8af0d9f562e1f8cfb4a76bd2ab5837fb5522ce90d945: Status 404 returned error can't find the container with id 537ba7c4c7e207a7d02e8af0d9f562e1f8cfb4a76bd2ab5837fb5522ce90d945 Jul 21 21:47:07.404974 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:07.404931 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-j8hjz_91f8a9a5-9cd0-4c99-ae2f-cc2c0c09fe8f/volume-data-source-validator/0.log" Jul 21 21:47:07.640061 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:07.640022 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" event={"ID":"fd4d768e-2159-4546-a387-a0db53507a10","Type":"ContainerStarted","Data":"44b01ed06c7c35f4b10d9bb4ce567deda637656504e3aad82084222d437361f9"} Jul 21 21:47:07.640260 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:07.640078 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" event={"ID":"fd4d768e-2159-4546-a387-a0db53507a10","Type":"ContainerStarted","Data":"537ba7c4c7e207a7d02e8af0d9f562e1f8cfb4a76bd2ab5837fb5522ce90d945"} Jul 21 21:47:07.640260 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:07.640111 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:07.660536 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:07.660443 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" podStartSLOduration=1.660428238 podStartE2EDuration="1.660428238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:47:07.658050049 +0000 UTC m=+2804.602671112" watchObservedRunningTime="2026-07-21 21:47:07.660428238 +0000 UTC m=+2804.605049304" Jul 21 21:47:08.528266 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:08.528232 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rk9dq_fe43a134-8bcb-41ac-8b93-f9d7f657f094/dns/0.log" Jul 21 21:47:08.551989 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:08.551961 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rk9dq_fe43a134-8bcb-41ac-8b93-f9d7f657f094/kube-rbac-proxy/0.log" Jul 21 21:47:08.631744 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:08.631711 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wk9cv_3e4e3cde-b184-49f6-bb67-1d5962d17492/dns-node-resolver/0.log" Jul 21 21:47:09.154398 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:09.154340 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85857f96-zd47z_f6a4a246-9f16-4f17-98a6-21ea631ca67a/registry/0.log" Jul 21 21:47:09.207196 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:09.207162 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7cgd8_8f71e36d-9129-487a-83bf-05ce9934d058/node-ca/0.log" Jul 21 21:47:10.405578 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:10.405542 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-clcpk_e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6/discovery/0.log" Jul 21 21:47:10.428961 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:10.428930 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b46444884-9dqzl_8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb/kube-auth-proxy/0.log" Jul 21 21:47:10.555695 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:10.555660 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hzfw4_54066cd0-970a-42c7-84a8-67bd17e03028/payload-pre-processing/0.log" Jul 21 21:47:10.584810 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:10.584772 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-g5x69_3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9/payload-processing/0.log" Jul 21 21:47:11.296529 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:11.296497 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5qq9n_666656f7-3729-44fa-a480-554e40f6e25c/serve-healthcheck-canary/0.log" Jul 21 21:47:11.824873 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:11.824836 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2rwnh_104d8201-c4ba-4d11-b47a-1db1a035dda7/insights-operator/0.log" Jul 21 21:47:11.825291 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:11.825270 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2rwnh_104d8201-c4ba-4d11-b47a-1db1a035dda7/insights-operator/1.log" Jul 21 21:47:11.929993 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:11.929961 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvcnx_f2b3f4b5-8578-4775-8ac9-b8cedd981d32/kube-rbac-proxy/0.log" Jul 21 21:47:11.973828 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:11.973804 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvcnx_f2b3f4b5-8578-4775-8ac9-b8cedd981d32/exporter/0.log" Jul 21 21:47:12.018354 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:12.018328 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvcnx_f2b3f4b5-8578-4775-8ac9-b8cedd981d32/extractor/0.log" Jul 21 21:47:13.656487 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:13.656458 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-g7mpp" Jul 21 21:47:14.319801 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.319763 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pvvx7_bf1921af-bb51-43c6-b544-cecff337f092/manager/0.log" Jul 21 21:47:14.510593 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.510562 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r2v84_de3fe3cc-480a-44ab-bf75-c0a5d46e2118/manager/0.log" Jul 21 21:47:14.619342 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.619242 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55d6556f9b-bf9cf_5258ce45-e3fe-4709-a616-63a72bd46073/manager/0.log" Jul 21 21:47:14.641109 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.641083 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ss4kl_0d1239b8-a55a-462c-a1ab-8c0a241c6ac6/server/0.log" Jul 21 21:47:14.711804 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.711776 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fkrdd_516b1e47-edcc-42dd-a976-019883307b6b/manager/0.log" Jul 21 21:47:14.849392 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.849352 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-rrjnd_5258cfd5-86d0-4592-b8bc-2ea861d1a35d/manager/0.log" Jul 21 21:47:14.869183 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:14.869157 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-rrjnd_5258cfd5-86d0-4592-b8bc-2ea861d1a35d/copy-manifests/0.log" Jul 21 21:47:16.649249 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:16.649211 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-clcpk_e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6/discovery/0.log" Jul 21 21:47:16.659858 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:16.659836 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b46444884-9dqzl_8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb/kube-auth-proxy/0.log" Jul 21 21:47:16.755994 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:16.755965 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hzfw4_54066cd0-970a-42c7-84a8-67bd17e03028/payload-pre-processing/0.log" Jul 21 21:47:16.773824 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:16.773800 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-g5x69_3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9/payload-processing/0.log" Jul 21 21:47:18.177204 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:18.177173 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-clcpk_e9991a6c-cdc1-4ccd-a9a3-e1c4f5adc4f6/discovery/0.log" Jul 21 21:47:18.193375 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:18.193343 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b46444884-9dqzl_8d5a1071-4ed8-4a2a-8f8b-1196b6f276bb/kube-auth-proxy/0.log" Jul 21 21:47:18.315711 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:18.315684 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hzfw4_54066cd0-970a-42c7-84a8-67bd17e03028/payload-pre-processing/0.log" Jul 21 21:47:18.336836 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:18.336804 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-g5x69_3fbe19d1-4ed9-4b12-8e49-eff86da1bcb9/payload-processing/0.log" Jul 21 21:47:21.767164 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:21.767136 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-mscck_60eb76a3-006a-44bc-814e-e5ed1fc6f42a/migrator/0.log" Jul 21 21:47:21.800889 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:21.800852 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-mscck_60eb76a3-006a-44bc-814e-e5ed1fc6f42a/graceful-termination/0.log" Jul 21 21:47:23.357980 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.357954 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/kube-multus-additional-cni-plugins/0.log" Jul 21 21:47:23.382803 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.382780 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/egress-router-binary-copy/0.log" Jul 21 21:47:23.405660 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.405633 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/cni-plugins/0.log" Jul 21 21:47:23.427121 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.427096 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/bond-cni-plugin/0.log" Jul 21 21:47:23.450986 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.450964 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/routeoverride-cni/0.log" Jul 21 21:47:23.482495 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.482462 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/whereabouts-cni-bincopy/0.log" Jul 21 21:47:23.521519 ip-10-0-132-65 kubenswrapper[2570]: I0721 21:47:23.521490 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwt58_3f1ce5f0-5af3-45e3-9751-43217e6e5946/whereabouts-cni/0.log"